[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:25:37 dist-test-slave-pk3j krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:25:37 dist-test-slave-pk3j krb5kdc[399](info): set up 2 sockets
Jun 20 14:25:37 dist-test-slave-pk3j krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:25:38.001542 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:25:38.109670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:38.210470 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:38.210728 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:38.210804 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:38.214633 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:38.214704 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:38.214737 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:38.214794 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:38.214829 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:38.219833 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:38.220707 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:38.221715 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:38.229409 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:38.229422 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:38.229418 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:38.229743 412 server_base.cc:1034] running on GCE node
I20260620 14:25:38.230329 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:38.231243 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:38.232429 412 hybrid_clock.cc:648] HybridClock initialized: now 1781965538232385 us; error 62 us; skew 500 ppm
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:38.234889 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:25:38.235720 412 webserver.cc:460] Webserver started at http://127.0.98.62:42997/ using document root <none> and password file <none>
I20260620 14:25:38.236222 412 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:38.236316 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:38.236527 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:38.238431 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "5aa2ae000b3a48d69ba2213dc0283f93"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "86be6f7c82244c15677073e93c9f89c5"
server_key_iv: "196f358dce2bd869454ec5a9ed34f210"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.238993 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "5aa2ae000b3a48d69ba2213dc0283f93"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "86be6f7c82244c15677073e93c9f89c5"
server_key_iv: "196f358dce2bd869454ec5a9ed34f210"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.242218 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 14:25:38.244313 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:38.245380 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:25:38.245503 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "5aa2ae000b3a48d69ba2213dc0283f93"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "86be6f7c82244c15677073e93c9f89c5"
server_key_iv: "196f358dce2bd869454ec5a9ed34f210"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.245602 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:38.284773 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:38.287372 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:38.287559 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:38.293825 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34543
I20260620 14:25:38.293848 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34543 every 8 connection(s)
I20260620 14:25:38.294696 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:38.297156 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:38.302039 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93: Bootstrap starting.
I20260620 14:25:38.304258 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:38.304947 480 log.cc:824] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:38.305190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 412
I20260620 14:25:38.305415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:38.306663 392 external_mini_cluster.cc:1428] Setting key ac944556a80e663f4d5a59c316b5a3ef
I20260620 14:25:38.306833 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93: No bootstrap required, opened a new log
I20260620 14:25:38.311039 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2ae000b3a48d69ba2213dc0283f93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34543 } }
I20260620 14:25:38.311254 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:38.311331 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa2ae000b3a48d69ba2213dc0283f93, State: Initialized, Role: FOLLOWER
I20260620 14:25:38.311719 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2ae000b3a48d69ba2213dc0283f93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34543 } }
I20260620 14:25:38.311856 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:25:38.311936 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:25:38.312036 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:38.312812 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2ae000b3a48d69ba2213dc0283f93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34543 } }
I20260620 14:25:38.313117 480 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa2ae000b3a48d69ba2213dc0283f93; no voters:
I20260620 14:25:38.313421 480 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:25:38.313536 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:38.313737 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa2ae000b3a48d69ba2213dc0283f93, State: Running, Role: LEADER
I20260620 14:25:38.314029 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2ae000b3a48d69ba2213dc0283f93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34543 } }
I20260620 14:25:38.314567 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:38.315552 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aa2ae000b3a48d69ba2213dc0283f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2ae000b3a48d69ba2213dc0283f93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34543 } } }
I20260620 14:25:38.315713 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:38.316012 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa2ae000b3a48d69ba2213dc0283f93. Latest consensus state: current_term: 1 leader_uuid: "5aa2ae000b3a48d69ba2213dc0283f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa2ae000b3a48d69ba2213dc0283f93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34543 } } }
I20260620 14:25:38.316108 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:38.316382 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:38.319535 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:38.324942 494 catalog_manager.cc:1347] Generated new cluster ID: 6f66616d33e445d8a300a8824c176f5a
I20260620 14:25:38.325039 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:25:38.330843 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:38.309250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56430 (local address 127.0.98.62:34543)
0620 14:25:38.309632 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:38.309646 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:38.310811 (+ 1165us) server_negotiation.cc:407] Connection header received
0620 14:25:38.313727 (+ 2916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:38.313742 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:38.314082 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:38.314397 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:38.317315 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.318747 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.320399 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.321082 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.321263 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:38.325454 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:38.325460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:38.325476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:38.325508 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:38.327511 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:38.328091 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:38.328097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:38.328102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:38.328232 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:38.328565 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:38.328568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:38.328570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:38.329036 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:38.329247 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:38.329784 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 14:25:38.330045 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":111,"threads_started":1}
I20260620 14:25:38.337652 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:38.338593 494 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:38.344121 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa2ae000b3a48d69ba2213dc0283f93: Generated new TSK 0
I20260620 14:25:38.344684 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:25:38.370136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34543
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:38.473304 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:38.473521 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:38.473584 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:38.477630 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:38.477676 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:38.477777 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:25:38.482496 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:38.483402 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:38.484372 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:38.492972 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:38.493062 506 server_base.cc:1034] running on GCE node
W20260620 14:25:38.493096 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:38.492973 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:38.493569 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:38.494094 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:38.495244 506 hybrid_clock.cc:648] HybridClock initialized: now 1781965538495217 us; error 41 us; skew 500 ppm
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:38.497855 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:25:38.498543 506 webserver.cc:460] Webserver started at http://127.0.98.1:36581/ using document root <none> and password file <none>
I20260620 14:25:38.499008 506 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:38.499059 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:38.499212 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:38.501034 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "e294d26ed1d042828cf0e20046680ea0"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "4f64ab1b1e7fcf33ca39b16a52eb115e"
server_key_iv: "bd652945f67fa574092d02bba0efd6cf"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.501583 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e294d26ed1d042828cf0e20046680ea0"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "4f64ab1b1e7fcf33ca39b16a52eb115e"
server_key_iv: "bd652945f67fa574092d02bba0efd6cf"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.504876 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:25:38.507046 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:38.508117 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:38.508234 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "e294d26ed1d042828cf0e20046680ea0"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "4f64ab1b1e7fcf33ca39b16a52eb115e"
server_key_iv: "bd652945f67fa574092d02bba0efd6cf"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.508327 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:38.541040 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:38.543802 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:38.543932 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:38.544392 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:38.545142 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:38.545185 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:38.545389 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:38.545414 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:38.553511 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36521
I20260620 14:25:38.553586 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36521 every 8 connection(s)
I20260620 14:25:38.554365 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:25:38.555238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 506
I20260620 14:25:38.555312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:25:38.555575 392 external_mini_cluster.cc:1428] Setting key 654e81313455e519e0139b4078c13b74
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:25:38.568533 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:38.556092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49657 (local address 127.0.98.62:34543)
0620 14:25:38.556213 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:38.556218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:38.557105 (+ 887us) server_negotiation.cc:407] Connection header received
0620 14:25:38.558419 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:38.558423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:38.558480 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:38.558575 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:38.559657 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.560906 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.562197 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.562622 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.562763 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:38.564883 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:38.564888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:38.564893 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:38.564928 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:38.566493 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:38.566947 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:38.566949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:38.566950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:38.567053 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:38.567370 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:38.567373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:38.567374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:38.567638 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:38.567764 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:38.568313 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 14:25:38.568417 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:25:38.569619 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:38.556490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34543 (local address 127.0.98.1:49657)
0620 14:25:38.556965 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:38.557012 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:38.558219 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:38.558738 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:38.558745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:38.559068 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:38.559513 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:38.559525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.561055 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.561058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:38.562052 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:38.562062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.562775 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.562777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:38.562904 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:38.563435 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:38.563449 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:38.564725 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:38.566599 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:38.566603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:38.566614 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:38.566850 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:38.567152 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:38.567155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:38.567157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:38.567286 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:38.567787 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:38.567795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:38.568053 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 14:25:38.568265 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 14:25:38.568602 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":137,"threads_started":1}
I20260620 14:25:38.570739 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34543
I20260620 14:25:38.570969 635 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:38.571419 635 heartbeater.cc:511] Master 127.0.98.62:34543 requested a full tablet report, sending...
I20260620 14:25:38.572841 444 ts_manager.cc:194] Registered new tserver with Master: e294d26ed1d042828cf0e20046680ea0 (127.0.98.1:36521)
I20260620 14:25:38.573968 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:25:38.581094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34543
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:38.685233 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:38.685460 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:38.685499 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:38.689285 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:38.689368 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:38.689474 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:25:38.694365 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:38.695230 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:38.696178 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:38.705044 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:38.705112 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:38.705253 640 server_base.cc:1034] running on GCE node
W20260620 14:25:38.705060 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:38.705710 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:38.706323 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:38.707507 640 hybrid_clock.cc:648] HybridClock initialized: now 1781965538707496 us; error 25 us; skew 500 ppm
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:38.710011 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:25:38.710829 640 webserver.cc:460] Webserver started at http://127.0.98.2:46619/ using document root <none> and password file <none>
I20260620 14:25:38.711345 640 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:38.711437 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:38.711645 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:38.713629 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ae96a5a8185437797ce05d95fb31afe"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "1548354c72fe07930ea85ee63695e03b"
server_key_iv: "66c09b16c7dff0df05849ee145ed4f61"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.714206 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ae96a5a8185437797ce05d95fb31afe"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "1548354c72fe07930ea85ee63695e03b"
server_key_iv: "66c09b16c7dff0df05849ee145ed4f61"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.717917 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:25:38.720156 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:38.721418 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:25:38.721531 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "0ae96a5a8185437797ce05d95fb31afe"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "1548354c72fe07930ea85ee63695e03b"
server_key_iv: "66c09b16c7dff0df05849ee145ed4f61"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.721634 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:38.731549 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:38.734087 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:38.734287 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:38.734809 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:38.735560 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:38.735628 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:38.735693 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:38.735728 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:38.744716 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41417
I20260620 14:25:38.744745 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41417 every 8 connection(s)
I20260620 14:25:38.745580 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:25:38.746500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 640
I20260620 14:25:38.746625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:25:38.746940 392 external_mini_cluster.cc:1428] Setting key 3f621f6658d42db9248274cc1cbfca11
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:38.763095 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:38.747333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58789 (local address 127.0.98.62:34543)
0620 14:25:38.747480 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:38.747485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:38.748342 (+ 857us) server_negotiation.cc:407] Connection header received
0620 14:25:38.750256 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:38.750262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:38.750321 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:38.750427 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:38.751440 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.751960 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.753049 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.753332 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.753447 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:38.759335 (+ 5888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:38.759341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:38.759349 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:38.759372 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:38.761022 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:38.761635 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:38.761637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:38.761639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:38.761709 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:38.762084 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:38.762086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:38.762087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:38.762241 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:38.762342 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:38.762799 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 14:25:38.762927 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:25:38.763801 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:38.747719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34543 (local address 127.0.98.2:58789)
0620 14:25:38.748184 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:38.748246 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:38.750018 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:38.750543 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:38.750550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:38.750872 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:38.751308 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:38.751321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.752083 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.752086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:38.752923 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:38.752933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.753480 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.753484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:38.753642 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:38.757420 (+ 3778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:38.757435 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:38.759178 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:38.761141 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:38.761150 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:38.761166 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:38.761542 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:38.761797 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:38.761803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:38.761808 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:38.761989 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:38.762344 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:38.762352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:38.762572 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 14:25:38.762747 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 14:25:38.763040 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":96,"threads_started":1}
I20260620 14:25:38.764869 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34543
I20260620 14:25:38.765110 769 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:38.765573 769 heartbeater.cc:511] Master 127.0.98.62:34543 requested a full tablet report, sending...
I20260620 14:25:38.766609 444 ts_manager.cc:194] Registered new tserver with Master: 0ae96a5a8185437797ce05d95fb31afe (127.0.98.2:41417)
I20260620 14:25:38.767160 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:25:38.772069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34543
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:38.878505 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:38.878716 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:38.878752 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:38.882661 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:38.882709 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:38.882766 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:25:38.887709 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:38.888599 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:38.889539 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:38.898335 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:38.898396 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:38.898335 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:38.898499 774 server_base.cc:1034] running on GCE node
I20260620 14:25:38.899003 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:38.899554 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:38.900734 774 hybrid_clock.cc:648] HybridClock initialized: now 1781965538900724 us; error 35 us; skew 500 ppm
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:38.903818 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:25:38.904610 774 webserver.cc:460] Webserver started at http://127.0.98.3:43883/ using document root <none> and password file <none>
I20260620 14:25:38.905164 774 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:38.905270 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:38.905512 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:38.907457 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "94ef049b0d3e4060bcdc60bab3e41bfc"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "6e71c67733a561842acf3eeb3502f588"
server_key_iv: "1cae8d5d95c3da3150b991ba64c07b78"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.908000 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "94ef049b0d3e4060bcdc60bab3e41bfc"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "6e71c67733a561842acf3eeb3502f588"
server_key_iv: "1cae8d5d95c3da3150b991ba64c07b78"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.911432 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 14:25:38.913681 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:38.914896 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 14:25:38.915019 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "94ef049b0d3e4060bcdc60bab3e41bfc"
format_stamp: "Formatted at 2026-06-20 14:25:38 on dist-test-slave-pk3j"
server_key: "6e71c67733a561842acf3eeb3502f588"
server_key_iv: "1cae8d5d95c3da3150b991ba64c07b78"
server_key_version: "encryptionkey@0"
I20260620 14:25:38.915154 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:38.933516 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:38.936044 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:38.936223 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:38.936735 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:38.937526 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:38.937589 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:38.937668 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:38.937711 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:38.946709 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35349
I20260620 14:25:38.946739 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35349 every 8 connection(s)
I20260620 14:25:38.947505 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:38.957608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 774
I20260620 14:25:38.957690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:25:38.957983 392 external_mini_cluster.cc:1428] Setting key 445bec5d198f4bae00e514c11f28dfa2
I20260620 14:25:38.959873 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:38.949089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45749 (local address 127.0.98.62:34543)
0620 14:25:38.949266 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:38.949271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:38.950051 (+ 780us) server_negotiation.cc:407] Connection header received
0620 14:25:38.951240 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:38.951243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:38.951287 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:38.951355 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:38.952326 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.952807 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.954049 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.954258 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.954345 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:38.956521 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:38.956524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:38.956526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:38.956551 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:38.957875 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:38.958382 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:38.958384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:38.958385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:38.958482 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:38.958861 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:38.958865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:38.958867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:38.959082 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:38.959186 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:38.959610 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 14:25:38.959717 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:25:38.960534 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:38.949528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34543 (local address 127.0.98.3:45749)
0620 14:25:38.949912 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:38.949966 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:38.951057 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:38.951474 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:38.951480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:38.951794 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:38.952206 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:38.952217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.952961 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.952964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:38.953942 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:38.953952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.954384 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.954386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:38.954511 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:38.954971 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:38.954985 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:38.956378 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:38.957972 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:38.957977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:38.957991 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:38.958291 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:38.958571 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:38.958587 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:38.958592 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:38.958760 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:38.959182 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:38.959187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:38.959400 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 14:25:38.959567 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:25:38.959831 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260620 14:25:38.961459 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34543
I20260620 14:25:38.961664 903 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:38.962069 903 heartbeater.cc:511] Master 127.0.98.62:34543 requested a full tablet report, sending...
I20260620 14:25:38.962934 444 ts_manager.cc:194] Registered new tserver with Master: 94ef049b0d3e4060bcdc60bab3e41bfc (127.0.98.3:35349)
I20260620 14:25:38.963394 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45749
I20260620 14:25:38.971846 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:25:38.994645 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:38.986185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.0.98.1:36521)
0620 14:25:38.986473 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:38.986485 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:38.986510 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:25:38.986564 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:38.986576 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:38.986725 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:38.986887 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:38.987492 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.988237 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.989017 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:38.989407 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:38.989521 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:38.991662 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:38.991670 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:38.991680 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:38.991701 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:38.992986 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:38.993448 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:38.993453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:38.993454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:38.993522 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:38.993801 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:38.993804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:38.993806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:38.994005 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:38.994190 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:38.994366 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:25:38.994497 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
W20260620 14:25:38.995433 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35624
Jun 20 14:25:38 dist-test-slave-pk3j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:25:39 dist-test-slave-pk3j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:25:39.017978 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:39.010297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.0.98.1:36521)
0620 14:25:39.010429 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:39.010432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:39.010482 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:25:39.010567 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:39.010569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:39.010602 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:39.010666 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:39.011284 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:39.011751 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:39.012555 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:39.012770 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:39.012842 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:39.014995 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:39.015001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:39.015008 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:39.015027 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:39.016309 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:39.016740 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:39.016745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:39.016747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:39.016822 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:39.017093 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:39.017098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:39.017099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:39.017315 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:39.017444 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:39.017701 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 14:25:39.017814 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:25:39.018822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 506
I20260620 14:25:39.024737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 640
I20260620 14:25:39.031829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 774
I20260620 14:25:39.036772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 412
2026-06-20T14:25:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2057 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:25:39 dist-test-slave-pk3j krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:25:39 dist-test-slave-pk3j krb5kdc[917](info): set up 2 sockets
Jun 20 14:25:39 dist-test-slave-pk3j krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 14:25:40.028648 392 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:25:40 dist-test-slave-pk3j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:25:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:25:40.135725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:40.241819 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:40.242022 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:40.242056 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:40.245813 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:40.245862 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:40.245882 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:40.245898 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:40.245913 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:40.250528 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:40.251293 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:40.252110 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:40.259941 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:40.259958 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:40.259943 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:40.260139 930 server_base.cc:1034] running on GCE node
I20260620 14:25:40.260658 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:40.261678 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:40.262854 930 hybrid_clock.cc:648] HybridClock initialized: now 1781965540262827 us; error 40 us; skew 500 ppm
Jun 20 14:25:40 dist-test-slave-pk3j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:40.265383 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:25:40.266252 930 webserver.cc:460] Webserver started at http://127.0.98.62:37273/ using document root <none> and password file <none>
I20260620 14:25:40.266777 930 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:40.266877 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:40.267092 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:40.268898 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "2b22b5cbbe6f43fda0d4befcbbe06657"
format_stamp: "Formatted at 2026-06-20 14:25:40 on dist-test-slave-pk3j"
server_key: "80bb0ced06b15d22359e39accd6e7715"
server_key_iv: "8dc9d766150c82efe9c19bc2ddcca2a2"
server_key_version: "encryptionkey@0"
I20260620 14:25:40.269436 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b22b5cbbe6f43fda0d4befcbbe06657"
format_stamp: "Formatted at 2026-06-20 14:25:40 on dist-test-slave-pk3j"
server_key: "80bb0ced06b15d22359e39accd6e7715"
server_key_iv: "8dc9d766150c82efe9c19bc2ddcca2a2"
server_key_version: "encryptionkey@0"
I20260620 14:25:40.272763 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:25:40.274967 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:40.276160 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:25:40.276252 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "2b22b5cbbe6f43fda0d4befcbbe06657"
format_stamp: "Formatted at 2026-06-20 14:25:40 on dist-test-slave-pk3j"
server_key: "80bb0ced06b15d22359e39accd6e7715"
server_key_iv: "8dc9d766150c82efe9c19bc2ddcca2a2"
server_key_version: "encryptionkey@0"
I20260620 14:25:40.276326 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:40.294297 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:40.296957 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:40.297091 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:40.303275 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43741
I20260620 14:25:40.303355 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43741 every 8 connection(s)
I20260620 14:25:40.304083 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:40.306635 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:40.310839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 930
I20260620 14:25:40.310925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:40.311154 392 external_mini_cluster.cc:1428] Setting key aa9126c72c9b77081fb41386e7445d3f
I20260620 14:25:40.311611 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657: Bootstrap starting.
I20260620 14:25:40.314167 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:40.314970 998 log.cc:824] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:40.316828 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657: No bootstrap required, opened a new log
Jun 20 14:25:40 dist-test-slave-pk3j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:40.320673 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22b5cbbe6f43fda0d4befcbbe06657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } }
I20260620 14:25:40.320936 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:40.321009 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b22b5cbbe6f43fda0d4befcbbe06657, State: Initialized, Role: FOLLOWER
I20260620 14:25:40.321494 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22b5cbbe6f43fda0d4befcbbe06657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } }
I20260620 14:25:40.321646 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:25:40.321743 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:25:40.321832 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:40.322858 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22b5cbbe6f43fda0d4befcbbe06657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } }
I20260620 14:25:40.323225 998 leader_election.cc:302] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b22b5cbbe6f43fda0d4befcbbe06657; no voters:
I20260620 14:25:40.323523 998 leader_election.cc:288] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:25:40.323614 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:40.323838 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [term 1 LEADER]: Becoming Leader. State: Replica: 2b22b5cbbe6f43fda0d4befcbbe06657, State: Running, Role: LEADER
I20260620 14:25:40.324184 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22b5cbbe6f43fda0d4befcbbe06657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } }
I20260620 14:25:40.324720 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:40.325639 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b22b5cbbe6f43fda0d4befcbbe06657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22b5cbbe6f43fda0d4befcbbe06657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } } }
I20260620 14:25:40.325762 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:40.312300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.0.98.62:43741)
0620 14:25:40.312683 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:40.312701 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:40.312745 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:25:40.313538 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:40.313560 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:40.313925 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:40.314273 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:40.314891 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:40.316257 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:40.317140 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:40.317804 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:40.317961 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:40.320207 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:40.320212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:40.320235 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:40.320262 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:40.322199 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:40.322594 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:40.322599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:40.322605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:40.322753 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:40.323036 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:40.323039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:40.323040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:40.323555 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:40.324379 (+ 824us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:40.324514 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:25:40.324773 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260620 14:25:40.325858 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:40.325949 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b22b5cbbe6f43fda0d4befcbbe06657. Latest consensus state: current_term: 1 leader_uuid: "2b22b5cbbe6f43fda0d4befcbbe06657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b22b5cbbe6f43fda0d4befcbbe06657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } } }
I20260620 14:25:40.326320 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:40.327235 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:40.329921 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:25:40.334668 1013 catalog_manager.cc:1347] Generated new cluster ID: 81f39f6b9c3f4656ae67bfaf39f223ae
I20260620 14:25:40.334735 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:25:40.340751 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:40.341833 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:40.352103 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b22b5cbbe6f43fda0d4befcbbe06657: Generated new TSK 0
I20260620 14:25:40.352759 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:25:40.376138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:40.477617 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:40.477826 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:40.477905 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:40.481765 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:40.481808 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:40.481910 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:25:40.486418 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:40.487270 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:40.488173 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:40.496531 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:40.496568 1024 server_base.cc:1034] running on GCE node
W20260620 14:25:40.496536 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:40.496536 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:40.497126 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:40.497668 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:40.498852 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781965540498836 us; error 24 us; skew 500 ppm
Jun 20 14:25:40 dist-test-slave-pk3j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:40.501242 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:25:40.501933 1024 webserver.cc:460] Webserver started at http://127.0.98.1:38897/ using document root <none> and password file <none>
I20260620 14:25:40.502417 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:40.502496 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:40.502693 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:40.504377 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f"
format_stamp: "Formatted at 2026-06-20 14:25:40 on dist-test-slave-pk3j"
server_key: "c6d2ba4a89390c0e711a1ddcf2a8bb22"
server_key_iv: "1a7069669efe418ca03014fd703063b3"
server_key_version: "encryptionkey@0"
I20260620 14:25:40.504863 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f"
format_stamp: "Formatted at 2026-06-20 14:25:40 on dist-test-slave-pk3j"
server_key: "c6d2ba4a89390c0e711a1ddcf2a8bb22"
server_key_iv: "1a7069669efe418ca03014fd703063b3"
server_key_version: "encryptionkey@0"
I20260620 14:25:40.508105 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:25:40.510238 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:40.511250 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:25:40.511368 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f"
format_stamp: "Formatted at 2026-06-20 14:25:40 on dist-test-slave-pk3j"
server_key: "c6d2ba4a89390c0e711a1ddcf2a8bb22"
server_key_iv: "1a7069669efe418ca03014fd703063b3"
server_key_version: "encryptionkey@0"
I20260620 14:25:40.511458 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:40.522094 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:40.524663 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:40.524837 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:40.525449 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:40.526250 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:40.526316 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:40.526378 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:40.526428 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:40.534503 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38711
I20260620 14:25:40.534528 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38711 every 8 connection(s)
I20260620 14:25:40.535223 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:25:40.540943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 1024
I20260620 14:25:40.541056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:25:40.541357 392 external_mini_cluster.cc:1428] Setting key ecf89060a31326245b3037f6d8829108
Jun 20 14:25:40 dist-test-slave-pk3j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:40.548308 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:40.536758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49259 (local address 127.0.98.62:43741)
0620 14:25:40.536913 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:40.536918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:40.537691 (+ 773us) server_negotiation.cc:407] Connection header received
0620 14:25:40.538859 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:40.538862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:40.538904 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:40.538975 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:40.539904 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:40.540437 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:40.541662 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:40.541938 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:40.542037 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:40.544678 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:40.544681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:40.544683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:40.544702 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:40.546404 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:40.546885 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:40.546890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:40.546894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:40.546978 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:40.547252 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:40.547257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:40.547259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:40.547479 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:40.547588 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:40.547972 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 14:25:40.548094 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:25:40.548889 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:40.537105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43741 (local address 127.0.98.1:49259)
0620 14:25:40.537530 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:40.537566 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:40.538679 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:40.539115 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:40.539121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:40.539407 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:40.539792 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:40.539803 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:40.540534 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:40.540536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:40.541550 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:40.541558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:40.542068 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:40.542072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:40.542232 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:40.542793 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:40.542811 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:40.544504 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:40.546523 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:40.546527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:40.546538 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:40.546776 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:40.547063 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:40.547065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:40.547066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:40.547168 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:40.547589 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:40.547593 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:40.547770 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 14:25:40.547898 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:25:40.548089 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":145,"threads_started":1}
I20260620 14:25:40.549968 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43741
I20260620 14:25:40.550179 1153 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:40.550588 1153 heartbeater.cc:511] Master 127.0.98.62:43741 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:25:40.551945 956 ts_manager.cc:194] Registered new tserver with Master: bdb1218f5b9f42cb89f8cc0bcf5a5a3f (127.0.98.1:38711)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:25:40.553117 956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:25:40.566125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:40.666283 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:40.666486 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:40.666571 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:40.670410 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:40.670454 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:40.670552 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:25:40.675263 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:40.676173 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:40.677069 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:40.685949 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:40.685940 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:40.686002 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:40.686194 1158 server_base.cc:1034] running on GCE node
I20260620 14:25:40.686654 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:40.687320 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:40.688500 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781965540688488 us; error 28 us; skew 500 ppm
Jun 20 14:25:40 dist-test-slave-pk3j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:40.691284 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:25:40.692116 1158 webserver.cc:460] Webserver started at http://127.0.98.2:39221/ using document root <none> and password file <none>
I20260620 14:25:40.692667 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:40.692730 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:40.692874 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:40.694895 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "bd79118ced6f4363b70d22fb5eca505f"
format_stamp: "Formatted at 2026-06-20 14:25:40 on dist-test-slave-pk3j"
server_key: "d19baaf735719d20abc104ea9148ce7e"
server_key_iv: "7e11053ae8d1c6604081e84f29612121"
server_key_version: "encryptionkey@0"
I20260620 14:25:40.695453 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bd79118ced6f4363b70d22fb5eca505f"
format_stamp: "Formatted at 2026-06-20 14:25:40 on dist-test-slave-pk3j"
server_key: "d19baaf735719d20abc104ea9148ce7e"
server_key_iv: "7e11053ae8d1c6604081e84f29612121"
server_key_version: "encryptionkey@0"
I20260620 14:25:40.699080 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:25:40.701709 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:40.702901 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:25:40.703011 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "bd79118ced6f4363b70d22fb5eca505f"
format_stamp: "Formatted at 2026-06-20 14:25:40 on dist-test-slave-pk3j"
server_key: "d19baaf735719d20abc104ea9148ce7e"
server_key_iv: "7e11053ae8d1c6604081e84f29612121"
server_key_version: "encryptionkey@0"
I20260620 14:25:40.703102 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:40.724081 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:40.727240 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:40.727416 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:40.727905 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:40.728618 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:40.728684 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:40.728749 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:40.728797 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:40.737402 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36929
I20260620 14:25:40.737449 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36929 every 8 connection(s)
I20260620 14:25:40.738204 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:25:40.741400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 1158
I20260620 14:25:40.741528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:25:40.741806 392 external_mini_cluster.cc:1428] Setting key fbb180dd1f5bb70a81eb2ec0bb62e454
Jun 20 14:25:40 dist-test-slave-pk3j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:25:40.750803 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:40.739814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58577 (local address 127.0.98.62:43741)
0620 14:25:40.739956 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:40.739960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:40.740624 (+ 664us) server_negotiation.cc:407] Connection header received
0620 14:25:40.742070 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:40.742074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:40.742132 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:40.742235 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:40.743432 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:40.743936 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:40.745013 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:40.745284 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:40.745393 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:40.747537 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:40.747541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:40.747543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:40.747570 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:40.748831 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:40.749350 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:40.749358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:40.749359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:40.749433 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:40.749792 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:40.749796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:40.749798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:40.749989 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:40.750081 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:40.750521 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 14:25:40.750631 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:25:40.751423 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:40.740102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43741 (local address 127.0.98.2:58577)
0620 14:25:40.740485 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:40.740526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:40.741830 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:40.742411 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:40.742419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:40.742771 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:40.743296 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:40.743309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:40.744070 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:40.744073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:40.744889 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:40.744897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:40.745418 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:40.745422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:40.745575 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:40.746110 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:40.746125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:40.747392 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:40.748962 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:40.748966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:40.748978 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:40.749254 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:40.749561 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:40.749565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:40.749572 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:40.749697 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:40.750076 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:40.750081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:40.750294 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 14:25:40.750459 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 14:25:40.750731 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
I20260620 14:25:40.752337 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43741
I20260620 14:25:40.752547 1287 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:40.752962 1287 heartbeater.cc:511] Master 127.0.98.62:43741 requested a full tablet report, sending...
I20260620 14:25:40.754477 956 ts_manager.cc:194] Registered new tserver with Master: bd79118ced6f4363b70d22fb5eca505f (127.0.98.2:36929)
I20260620 14:25:40.755002 956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:25:41.555033 1153 heartbeater.cc:503] Master 127.0.98.62:43741 was elected leader, sending a full tablet report...
I20260620 14:25:41.757191 1287 heartbeater.cc:503] Master 127.0.98.62:43741 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260620 14:25:41.834545 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.091s user 0.002s sys 0.001s
I20260620 14:25:41.835021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:41.945039 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:41.945259 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:41.945299 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:41.949012 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:41.949057 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:41.949111 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:25:41.953881 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:41.954732 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:41.955574 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:41.964879 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:41.964912 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:41.964923 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:41.965073 1292 server_base.cc:1034] running on GCE node
I20260620 14:25:41.965768 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:41.966315 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:41.967495 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781965541967478 us; error 36 us; skew 500 ppm
Jun 20 14:25:41 dist-test-slave-pk3j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:41.970135 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:25:41.970912 1292 webserver.cc:460] Webserver started at http://127.0.98.3:46155/ using document root <none> and password file <none>
I20260620 14:25:41.971429 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:41.971514 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:41.971726 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:41.973631 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "75894bd4226149db8aa937762642e4cf"
format_stamp: "Formatted at 2026-06-20 14:25:41 on dist-test-slave-pk3j"
server_key: "12926d3fde55b23698077ecabd1768e3"
server_key_iv: "1b2116454a10fded2209a75a6aa7feab"
server_key_version: "encryptionkey@0"
I20260620 14:25:41.974153 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "75894bd4226149db8aa937762642e4cf"
format_stamp: "Formatted at 2026-06-20 14:25:41 on dist-test-slave-pk3j"
server_key: "12926d3fde55b23698077ecabd1768e3"
server_key_iv: "1b2116454a10fded2209a75a6aa7feab"
server_key_version: "encryptionkey@0"
I20260620 14:25:41.977890 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:25:41.980391 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:41.981689 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:25:41.981817 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "75894bd4226149db8aa937762642e4cf"
format_stamp: "Formatted at 2026-06-20 14:25:41 on dist-test-slave-pk3j"
server_key: "12926d3fde55b23698077ecabd1768e3"
server_key_iv: "1b2116454a10fded2209a75a6aa7feab"
server_key_version: "encryptionkey@0"
I20260620 14:25:41.981926 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:41.999990 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:42.002625 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:42.002806 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:42.003383 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:42.004112 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:42.004182 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:42.004247 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:42.004300 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:42.013123 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32971
I20260620 14:25:42.013149 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32971 every 8 connection(s)
I20260620 14:25:42.013980 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:25:42.020269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 1292
I20260620 14:25:42.020359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:25:42.020644 392 external_mini_cluster.cc:1428] Setting key 38b84715f47f981cb22d54e0973d42c9
Jun 20 14:25:42 dist-test-slave-pk3j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:42.027034 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.015567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60505 (local address 127.0.98.62:43741)
0620 14:25:42.015790 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:42.015794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:42.016474 (+ 680us) server_negotiation.cc:407] Connection header received
0620 14:25:42.017823 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:42.017827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:42.017890 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:42.017997 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:42.019034 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.019583 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.020815 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.021048 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.021154 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.023537 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:42.023567 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:42.023569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:42.023591 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:42.024832 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:42.025396 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:42.025401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:42.025403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:42.025530 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:42.025856 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:42.025859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:42.025861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:42.026057 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:42.026158 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:42.026634 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 14:25:42.026771 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260620 14:25:42.027624 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.015944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43741 (local address 127.0.98.3:60505)
0620 14:25:42.016329 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:42.016366 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:42.017611 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:42.018165 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:42.018171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:42.018475 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:42.018881 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.018892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.019701 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.019705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.020657 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.020666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.021183 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.021186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.021373 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.021904 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:42.021919 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:42.023318 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:42.024947 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:42.024952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:42.024964 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:42.025292 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:42.025626 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:42.025629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:42.025631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:42.025758 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:42.026165 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:42.026170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:42.026397 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 14:25:42.026541 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 14:25:42.026851 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
I20260620 14:25:42.028586 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43741
I20260620 14:25:42.028795 1421 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:42.029310 1421 heartbeater.cc:511] Master 127.0.98.62:43741 requested a full tablet report, sending...
I20260620 14:25:42.030261 956 ts_manager.cc:194] Registered new tserver with Master: 75894bd4226149db8aa937762642e4cf (127.0.98.3:32971)
I20260620 14:25:42.030786 956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60505
I20260620 14:25:42.034133 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:25:42.043630 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.037300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.0.98.62:43741)
0620 14:25:42.037441 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:42.037445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:42.037632 (+ 187us) server_negotiation.cc:407] Connection header received
0620 14:25:42.037698 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:42.037700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:42.037731 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:42.037790 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:42.038405 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.038866 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.039617 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.039806 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.039874 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.041077 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:42.041080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:42.041081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:42.041097 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:42.042262 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:42.042613 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:42.042615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:42.042616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:42.042676 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:42.042897 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:42.042899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:42.042900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:42.043061 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:42.043156 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:42.043353 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 14:25:42.043441 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:25:42.047628 956 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:25:42.050506 956 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:25:42.063875 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.058154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36929 (local address 127.0.0.1:60002)
0620 14:25:42.058668 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:42.058730 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:42.058842 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:42.059567 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:42.059572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:42.059594 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:42.059699 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.059706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.060924 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.060927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.062419 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.062428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.063377 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.063381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.063481 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.063518 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:25:42.063626 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:25:42.063734 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":121,"threads_started":1}
I20260620 14:25:42.063890 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.058320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.0.98.2:36929)
0620 14:25:42.059098 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:42.059105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:42.059126 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:25:42.059198 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:42.059203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:42.059405 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:42.059571 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:42.059829 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.060815 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.062540 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.063242 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.063366 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.063501 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:42.063660 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:25:42.063746 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":94,"threads_started":1}
I20260620 14:25:42.064230 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.058351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32971 (local address 127.0.0.1:42964)
0620 14:25:42.059461 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:42.059475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:42.059559 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:42.060040 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:42.060043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:42.060059 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:42.060132 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.060138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.061484 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.061487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.062589 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.062596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.063754 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.063760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.063856 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.063875 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:25:42.064019 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 14:25:42.064084 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":23,"thread_start_us":89,"threads_started":1}
I20260620 14:25:42.064512 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.058989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.0.98.1:38711)
0620 14:25:42.059345 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:42.059352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:42.059899 (+ 547us) server_negotiation.cc:407] Connection header received
0620 14:25:42.060004 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:42.060012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:42.060173 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:42.060350 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:42.060521 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.061550 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.062920 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.063696 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.063795 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.063924 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:42.064299 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 14:25:42.064389 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":76,"threads_started":1}
I20260620 14:25:42.064607 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.058857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38711 (local address 127.0.0.1:49754)
0620 14:25:42.059790 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:42.059805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:42.059893 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:42.060342 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:42.060345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:42.060361 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:42.060431 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.060435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.061651 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.061656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.062794 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.062804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.063979 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.063985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.064076 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.064094 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:25:42.064159 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:25:42.064208 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":80,"threads_started":1}
I20260620 14:25:42.064630 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.058952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.0.98.3:32971)
0620 14:25:42.059334 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:42.059345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:42.059649 (+ 304us) server_negotiation.cc:407] Connection header received
0620 14:25:42.059719 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:42.059723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:42.059907 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:42.060053 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:42.060252 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.061357 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.062710 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.063449 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.063563 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.063688 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:42.064391 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 14:25:42.064469 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":99,"threads_started":1}
I20260620 14:25:42.066824 1221 tablet_service.cc:1467] Processing CreateTablet for tablet c6bda506b2544388932063bd5a1eb1c5 (DEFAULT_TABLE table=test-table [id=02b32d0323604663ba4801883bc2b8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:42.067042 1355 tablet_service.cc:1467] Processing CreateTablet for tablet c6bda506b2544388932063bd5a1eb1c5 (DEFAULT_TABLE table=test-table [id=02b32d0323604663ba4801883bc2b8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:42.067770 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bda506b2544388932063bd5a1eb1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:42.067909 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bda506b2544388932063bd5a1eb1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:42.070956 1087 tablet_service.cc:1467] Processing CreateTablet for tablet c6bda506b2544388932063bd5a1eb1c5 (DEFAULT_TABLE table=test-table [id=02b32d0323604663ba4801883bc2b8bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:42.071889 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6bda506b2544388932063bd5a1eb1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:42.074630 1440 tablet_bootstrap.cc:492] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f: Bootstrap starting.
I20260620 14:25:42.075074 1441 tablet_bootstrap.cc:492] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf: Bootstrap starting.
I20260620 14:25:42.076838 1440 tablet_bootstrap.cc:654] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:42.077174 1441 tablet_bootstrap.cc:654] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:42.077714 1440 log.cc:824] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:42.078433 1441 log.cc:824] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:42.079666 1440 tablet_bootstrap.cc:492] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f: No bootstrap required, opened a new log
I20260620 14:25:42.079850 1440 ts_tablet_manager.cc:1397] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:25:42.080303 1441 tablet_bootstrap.cc:492] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf: No bootstrap required, opened a new log
I20260620 14:25:42.080498 1441 ts_tablet_manager.cc:1397] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 14:25:42.083966 1440 raft_consensus.cc:348] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd79118ced6f4363b70d22fb5eca505f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36929 } } peers { permanent_uuid: "75894bd4226149db8aa937762642e4cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38711 } }
I20260620 14:25:42.084003 1441 raft_consensus.cc:348] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd79118ced6f4363b70d22fb5eca505f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36929 } } peers { permanent_uuid: "75894bd4226149db8aa937762642e4cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38711 } }
I20260620 14:25:42.084240 1441 raft_consensus.cc:374] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:42.084249 1440 raft_consensus.cc:374] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:42.084303 1440 raft_consensus.cc:729] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd79118ced6f4363b70d22fb5eca505f, State: Initialized, Role: FOLLOWER
I20260620 14:25:42.084335 1441 raft_consensus.cc:729] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75894bd4226149db8aa937762642e4cf, State: Initialized, Role: FOLLOWER
I20260620 14:25:42.084805 1441 consensus_queue.cc:260] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd79118ced6f4363b70d22fb5eca505f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36929 } } peers { permanent_uuid: "75894bd4226149db8aa937762642e4cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38711 } }
I20260620 14:25:42.084805 1440 consensus_queue.cc:260] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd79118ced6f4363b70d22fb5eca505f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36929 } } peers { permanent_uuid: "75894bd4226149db8aa937762642e4cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38711 } }
I20260620 14:25:42.085424 1421 heartbeater.cc:503] Master 127.0.98.62:43741 was elected leader, sending a full tablet report...
I20260620 14:25:42.085743 1441 ts_tablet_manager.cc:1428] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 14:25:42.085743 1440 ts_tablet_manager.cc:1428] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:25:42.085922 1443 tablet_bootstrap.cc:492] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f: Bootstrap starting.
I20260620 14:25:42.088182 1443 tablet_bootstrap.cc:654] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:42.088968 1443 log.cc:824] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:42.090727 1443 tablet_bootstrap.cc:492] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f: No bootstrap required, opened a new log
I20260620 14:25:42.090895 1443 ts_tablet_manager.cc:1397] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260620 14:25:42.094080 1443 raft_consensus.cc:348] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd79118ced6f4363b70d22fb5eca505f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36929 } } peers { permanent_uuid: "75894bd4226149db8aa937762642e4cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38711 } }
I20260620 14:25:42.094269 1443 raft_consensus.cc:374] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:42.094337 1443 raft_consensus.cc:729] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb1218f5b9f42cb89f8cc0bcf5a5a3f, State: Initialized, Role: FOLLOWER
I20260620 14:25:42.094753 1443 consensus_queue.cc:260] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd79118ced6f4363b70d22fb5eca505f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36929 } } peers { permanent_uuid: "75894bd4226149db8aa937762642e4cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38711 } }
I20260620 14:25:42.095577 1443 ts_tablet_manager.cc:1428] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:25:42.142150 1448 raft_consensus.cc:482] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:25:42.142354 1448 raft_consensus.cc:504] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd79118ced6f4363b70d22fb5eca505f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36929 } } peers { permanent_uuid: "75894bd4226149db8aa937762642e4cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38711 } }
I20260620 14:25:42.143442 1448 leader_election.cc:288] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd79118ced6f4363b70d22fb5eca505f (127.0.98.2:36929), 75894bd4226149db8aa937762642e4cf (127.0.98.3:32971)
I20260620 14:25:42.147281 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.143853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36403 (local address 127.0.98.3:32971)
0620 14:25:42.143963 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:42.143967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:42.144131 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:25:42.144272 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:42.144279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:42.144315 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:42.144389 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:42.144675 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.145143 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.146304 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.146704 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.146841 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.146899 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:42.147102 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 14:25:42.147167 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:25:42.147286 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.143706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39371 (local address 127.0.98.2:36929)
0620 14:25:42.143841 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:42.143847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:42.144088 (+ 241us) server_negotiation.cc:407] Connection header received
0620 14:25:42.144232 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:42.144235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:42.144284 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:42.144363 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:42.144661 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.145154 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.146294 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.146706 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.146834 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.146900 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:42.147108 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 14:25:42.147173 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:25:42.147310 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.143625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36929 (local address 127.0.98.1:39371)
0620 14:25:42.143982 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:42.143998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:42.144123 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:42.144373 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:42.144376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:42.144389 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:42.144552 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.144557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.145297 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.145301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.146171 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.146179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.146832 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.146836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.146983 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.147001 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:25:42.147091 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:25:42.147159 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":31,"spinlock_wait_cycles":6016,"thread_start_us":125,"threads_started":1}
I20260620 14:25:42.147320 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.143638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32971 (local address 127.0.98.1:36403)
0620 14:25:42.144041 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:42.144052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:42.144175 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:42.144407 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:42.144410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:42.144447 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:42.144573 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.144577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.145264 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.145270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.146183 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:42.146192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.146830 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.146833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:42.146987 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.147004 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:25:42.147088 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:25:42.147152 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":134,"spinlock_wait_cycles":27136,"thread_start_us":109,"threads_started":1}
I20260620 14:25:42.147825 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bda506b2544388932063bd5a1eb1c5" candidate_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd79118ced6f4363b70d22fb5eca505f" is_pre_election: true
I20260620 14:25:42.147974 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bda506b2544388932063bd5a1eb1c5" candidate_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75894bd4226149db8aa937762642e4cf" is_pre_election: true
I20260620 14:25:42.148140 1241 raft_consensus.cc:2454] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdb1218f5b9f42cb89f8cc0bcf5a5a3f in term 0.
I20260620 14:25:42.148275 1375 raft_consensus.cc:2454] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdb1218f5b9f42cb89f8cc0bcf5a5a3f in term 0.
I20260620 14:25:42.148644 1040 leader_election.cc:302] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd79118ced6f4363b70d22fb5eca505f, bdb1218f5b9f42cb89f8cc0bcf5a5a3f; no voters:
I20260620 14:25:42.148906 1448 raft_consensus.cc:2781] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:25:42.148985 1448 raft_consensus.cc:482] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:25:42.149027 1448 raft_consensus.cc:3037] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:42.230751 1448 raft_consensus.cc:504] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd79118ced6f4363b70d22fb5eca505f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36929 } } peers { permanent_uuid: "75894bd4226149db8aa937762642e4cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38711 } }
I20260620 14:25:42.231256 1448 leader_election.cc:288] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [CANDIDATE]: Term 1 election: Requested vote from peers bd79118ced6f4363b70d22fb5eca505f (127.0.98.2:36929), 75894bd4226149db8aa937762642e4cf (127.0.98.3:32971)
I20260620 14:25:42.231695 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bda506b2544388932063bd5a1eb1c5" candidate_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75894bd4226149db8aa937762642e4cf"
I20260620 14:25:42.231673 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6bda506b2544388932063bd5a1eb1c5" candidate_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd79118ced6f4363b70d22fb5eca505f"
I20260620 14:25:42.231845 1375 raft_consensus.cc:3037] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:42.231846 1241 raft_consensus.cc:3037] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:42.233541 1375 raft_consensus.cc:2454] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdb1218f5b9f42cb89f8cc0bcf5a5a3f in term 1.
I20260620 14:25:42.233639 1241 raft_consensus.cc:2454] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdb1218f5b9f42cb89f8cc0bcf5a5a3f in term 1.
I20260620 14:25:42.233911 1042 leader_election.cc:302] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75894bd4226149db8aa937762642e4cf, bdb1218f5b9f42cb89f8cc0bcf5a5a3f; no voters:
I20260620 14:25:42.234081 1448 raft_consensus.cc:2781] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:42.234297 1448 raft_consensus.cc:686] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [term 1 LEADER]: Becoming Leader. State: Replica: bdb1218f5b9f42cb89f8cc0bcf5a5a3f, State: Running, Role: LEADER
I20260620 14:25:42.234707 1448 consensus_queue.cc:237] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd79118ced6f4363b70d22fb5eca505f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36929 } } peers { permanent_uuid: "75894bd4226149db8aa937762642e4cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38711 } }
I20260620 14:25:42.238631 962 catalog_manager.cc:5495] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to bdb1218f5b9f42cb89f8cc0bcf5a5a3f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd79118ced6f4363b70d22fb5eca505f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75894bd4226149db8aa937762642e4cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdb1218f5b9f42cb89f8cc0bcf5a5a3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38711 } health_report { overall_health: HEALTHY } } }
W20260620 14:25:42.239986 1288 tablet.cc:2369] T c6bda506b2544388932063bd5a1eb1c5 P bd79118ced6f4363b70d22fb5eca505f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:25:42 dist-test-slave-pk3j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:25:42.258265 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.249746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.0.98.1:38711)
0620 14:25:42.249884 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:42.249889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:42.250053 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:25:42.250186 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:42.250190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:42.250235 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:42.250315 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:42.250986 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.251469 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.252297 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.252509 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.252619 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.254992 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:42.255000 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:42.255011 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:42.255031 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:42.256721 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:42.257096 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:42.257100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:42.257101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:42.257187 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:42.257465 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:42.257467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:42.257468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:42.257676 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:42.257812 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:42.257996 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:25:42.258098 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 14:25:42.264962 1422 tablet.cc:2369] T c6bda506b2544388932063bd5a1eb1c5 P 75894bd4226149db8aa937762642e4cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:25:42 dist-test-slave-pk3j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965542, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 14:25:42.286988 1154 tablet.cc:2369] T c6bda506b2544388932063bd5a1eb1c5 P bdb1218f5b9f42cb89f8cc0bcf5a5a3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:25:42 dist-test-slave-pk3j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965542, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:25:42.340409 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.331742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.0.98.1:38711)
0620 14:25:42.331885 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:42.331892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:42.332021 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:25:42.332140 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:42.332145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:42.332200 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:42.332292 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:42.332905 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.333770 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.334573 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.335034 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.335151 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.337588 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:42.337591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:42.337593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:42.337614 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:42.338817 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:42.339211 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:42.339216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:42.339218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:42.339291 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:42.339563 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:42.339568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:42.339570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:42.339740 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:42.339887 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:42.340148 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:25:42.340258 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 14:25:42 dist-test-slave-pk3j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:25:42 dist-test-slave-pk3j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:25:42.444820 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:42.436408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.0.98.1:38711)
0620 14:25:42.436571 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:42.436579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:42.436720 (+ 141us) server_negotiation.cc:407] Connection header received
0620 14:25:42.436858 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:42.436864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:42.436923 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:42.437007 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:42.437720 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.438607 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.439402 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:42.439856 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:42.439967 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:42.442308 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:42.442311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:42.442313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:42.442361 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:42.443550 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:42.443899 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:42.443901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:42.443902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:42.443964 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:42.444192 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:42.444194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:42.444195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:42.444323 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:42.444430 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:42.444624 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:25:42.444707 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 14:25:42.446476 1067 tablet_service.cc:2908] Rejecting scan request for tablet c6bda506b2544388932063bd5a1eb1c5: Uninitialized: clean time has not yet been initialized
I20260620 14:25:42.448450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 1024
I20260620 14:25:42.453900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 1158
I20260620 14:25:42.459045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 1292
I20260620 14:25:42.463980 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 930
2026-06-20T14:25:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3428 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:25:43 dist-test-slave-pk3j krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:25:43 dist-test-slave-pk3j krb5kdc[1465](info): set up 2 sockets
Jun 20 14:25:43 dist-test-slave-pk3j krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260620 14:25:44.768989 392 mini_kdc.cc:121] Time spent starting KDC: real 2.283s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 14:25:44.919857 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.151s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 14:25:45.094173 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.100s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/test-user.keytab.
W20260620 14:25:45.731966 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.638s user 0.001s sys 0.001s
Jun 20 14:25:45 dist-test-slave-pk3j krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:25:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260620 14:25:46.213075 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.461s user 0.000s sys 0.003s
I20260620 14:25:46.213585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46439
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:46.317813 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:46.318082 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:46.318157 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:46.321956 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:46.322029 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:25:46.322062 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:46.322101 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:46.322137 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:46.322171 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:46.327055 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:46.327922 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:46.328781 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:46.336402 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:46.336369 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:46.336416 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:46.336640 1478 server_base.cc:1034] running on GCE node
I20260620 14:25:46.337298 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:46.338231 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:46.339404 1478 hybrid_clock.cc:648] HybridClock initialized: now 1781965546339382 us; error 34 us; skew 500 ppm
Jun 20 14:25:46 dist-test-slave-pk3j krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:46.341835 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:25:46.342654 1478 webserver.cc:460] Webserver started at http://127.0.98.62:38137/ using document root <none> and password file <none>
I20260620 14:25:46.343150 1478 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:46.343233 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:46.343439 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:46.345247 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "a4b36902d2c945e6a5114fd0b3f2d94b"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "43643f1972c6b14e724380e6e429d345"
server_key_iv: "58764858554f3a646edd424b2cf75434"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.345801 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4b36902d2c945e6a5114fd0b3f2d94b"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "43643f1972c6b14e724380e6e429d345"
server_key_iv: "58764858554f3a646edd424b2cf75434"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.349079 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:25:46.351204 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:46.352262 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:46.352383 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "a4b36902d2c945e6a5114fd0b3f2d94b"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "43643f1972c6b14e724380e6e429d345"
server_key_iv: "58764858554f3a646edd424b2cf75434"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.352475 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:46.365124 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:46.367632 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:46.367807 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:46.374917 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46439
I20260620 14:25:46.374927 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46439 every 8 connection(s)
I20260620 14:25:46.375730 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:46.378340 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:46.379186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 1478
I20260620 14:25:46.379283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:46.379511 392 external_mini_cluster.cc:1428] Setting key 694e153358ec9b645869aaccce03f96f
I20260620 14:25:46.383769 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b: Bootstrap starting.
I20260620 14:25:46.386273 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:46.387143 1556 log.cc:824] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:25:46 dist-test-slave-pk3j krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:46.389176 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b: No bootstrap required, opened a new log
I20260620 14:25:46.392434 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b36902d2c945e6a5114fd0b3f2d94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46439 } }
I20260620 14:25:46.392618 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:46.392683 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b36902d2c945e6a5114fd0b3f2d94b, State: Initialized, Role: FOLLOWER
I20260620 14:25:46.393076 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:46.380729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.0.98.62:46439)
0620 14:25:46.381061 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:46.381074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:46.381102 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:25:46.381719 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:46.381734 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:46.382014 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:46.382249 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:46.383079 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.383941 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.384993 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.385465 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.385597 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:46.388404 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:46.388408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:46.388418 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:46.388441 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:46.390441 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:46.390907 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:46.390911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:46.390915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:46.391008 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:46.391274 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:46.391276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:46.391277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:46.391600 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:46.391728 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:46.392014 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 14:25:46.392284 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
I20260620 14:25:46.393080 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b36902d2c945e6a5114fd0b3f2d94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46439 } }
I20260620 14:25:46.393273 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:25:46.393335 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:25:46.393453 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:46.394349 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b36902d2c945e6a5114fd0b3f2d94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46439 } }
I20260620 14:25:46.394649 1556 leader_election.cc:302] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4b36902d2c945e6a5114fd0b3f2d94b; no voters:
I20260620 14:25:46.394948 1556 leader_election.cc:288] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:25:46.395095 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:46.395285 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [term 1 LEADER]: Becoming Leader. State: Replica: a4b36902d2c945e6a5114fd0b3f2d94b, State: Running, Role: LEADER
I20260620 14:25:46.395640 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b36902d2c945e6a5114fd0b3f2d94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46439 } }
I20260620 14:25:46.396082 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:46.397294 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4b36902d2c945e6a5114fd0b3f2d94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b36902d2c945e6a5114fd0b3f2d94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46439 } } }
I20260620 14:25:46.397276 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b36902d2c945e6a5114fd0b3f2d94b. Latest consensus state: current_term: 1 leader_uuid: "a4b36902d2c945e6a5114fd0b3f2d94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b36902d2c945e6a5114fd0b3f2d94b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46439 } } }
I20260620 14:25:46.397415 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:46.397415 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:46.398260 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:46.400624 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:25:46.405296 1572 catalog_manager.cc:1347] Generated new cluster ID: b03a1d15a3c04633a9404bc107f7feea
I20260620 14:25:46.405349 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:25:46.410354 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:46.411228 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:46.426664 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b: Generated new TSK 0
I20260620 14:25:46.427264 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:25:46.458465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46439
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:46.558678 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:46.558916 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:46.558990 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:46.562656 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:46.562726 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:46.562790 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:25:46.562863 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:25:46.567721 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:46.568578 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:46.569456 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:46.578248 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:46.578245 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:46.578245 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:46.578837 1582 server_base.cc:1034] running on GCE node
I20260620 14:25:46.579149 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:46.579630 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:46.580778 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781965546580757 us; error 35 us; skew 500 ppm
Jun 20 14:25:46 dist-test-slave-pk3j krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:46.583295 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:25:46.584064 1582 webserver.cc:460] Webserver started at http://127.0.98.1:39019/ using document root <none> and password file <none>
I20260620 14:25:46.584544 1582 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:46.584625 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:46.584828 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:46.586697 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "dfdb84b4aee94b7faa68ff55a5405174"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "9e0a93b4abebbf48d857390b14466402"
server_key_iv: "fe21d9bad71bd020e2b3f4dfa128b88f"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.587219 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dfdb84b4aee94b7faa68ff55a5405174"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "9e0a93b4abebbf48d857390b14466402"
server_key_iv: "fe21d9bad71bd020e2b3f4dfa128b88f"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.590687 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:25:46.592739 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:46.593891 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:25:46.594030 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "dfdb84b4aee94b7faa68ff55a5405174"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "9e0a93b4abebbf48d857390b14466402"
server_key_iv: "fe21d9bad71bd020e2b3f4dfa128b88f"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.594129 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:46.607755 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:46.610253 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:46.610436 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:46.611708 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:46.611776 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:46.611828 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:46.611879 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:25:46 dist-test-slave-pk3j krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:46.622253 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43985
I20260620 14:25:46.622335 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43985 every 8 connection(s)
I20260620 14:25:46.622998 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:25:46.623713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 1582
I20260620 14:25:46.623827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:25:46.624050 392 external_mini_cluster.cc:1428] Setting key b420b99e81c19562f27d13213e6c4e28
I20260620 14:25:46.624491 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:46.612300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52857 (local address 127.0.98.62:46439)
0620 14:25:46.612431 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:46.612436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:46.613280 (+ 844us) server_negotiation.cc:407] Connection header received
0620 14:25:46.614160 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:46.614166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:46.614222 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:46.614319 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:46.615499 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.616176 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.617323 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.617671 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.617789 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:46.620891 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:46.620895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:46.620898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:46.620923 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:46.622497 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:46.622964 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:46.622969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:46.622971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:46.623053 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:46.623386 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:46.623391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:46.623393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:46.623629 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:46.623745 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:46.624220 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 14:25:46.624334 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:25:46.625345 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:46.612661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46439 (local address 127.0.98.1:52857)
0620 14:25:46.613088 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:46.613155 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:46.613967 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:46.614492 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:46.614502 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:46.614914 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:46.615354 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:46.615366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.616283 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.616287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:46.617160 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:46.617169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.617817 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.617821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:46.618003 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:46.618521 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:46.618546 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:46.620739 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:46.622617 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:46.622622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:46.622635 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:46.622860 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:46.623167 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:46.623170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:46.623172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:46.623292 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:46.623742 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:46.623748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:46.623998 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 14:25:46.624174 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 14:25:46.624437 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":108,"threads_started":1}
I20260620 14:25:46.626583 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46439
I20260620 14:25:46.626730 1713 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:46.627266 1713 heartbeater.cc:511] Master 127.0.98.62:46439 requested a full tablet report, sending...
I20260620 14:25:46.628705 1509 ts_manager.cc:194] Registered new tserver with Master: dfdb84b4aee94b7faa68ff55a5405174 (127.0.98.1:43985)
I20260620 14:25:46.630026 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52857
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:25:46.636657 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:46.629774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56543 (local address 127.0.98.62:46439)
0620 14:25:46.629870 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:46.629873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:46.629883 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:25:46.629908 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:46.629910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:46.629938 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:46.629999 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:46.630485 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.631308 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.632296 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.632503 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.632611 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:46.633703 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:46.633709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:46.633715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:46.633744 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:46.635119 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:46.635514 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:46.635516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:46.635528 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:46.635628 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:46.635925 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:46.635931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:46.635936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:46.636099 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:46.636177 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:46.636480 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 14:25:46.636555 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:25:46.636906 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:46.629464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46439 (local address 127.0.98.1:56543)
0620 14:25:46.629715 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:46.629729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:46.629802 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:46.630072 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:46.630075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:46.630255 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:46.630359 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:46.630366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.631442 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.631445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:46.632181 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:46.632190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.632612 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.632615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:46.632701 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:46.633195 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:46.633232 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:46.633553 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:46.635240 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:46.635243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:46.635244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:46.635415 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:46.635749 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:46.635755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:46.635758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:46.635837 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:46.636197 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:46.636203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:46.636314 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 14:25:46.636436 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:25:46.636601 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:25:46.649678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46439
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:46.750208 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:46.750407 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:46.750445 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:46.754096 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:46.754151 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:46.754187 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:25:46.754217 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:25:46.758898 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:46.759652 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:46.760448 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:46.768707 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:46.768707 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:46.768832 1722 server_base.cc:1034] running on GCE node
W20260620 14:25:46.768975 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:46.769353 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:46.769845 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:46.770996 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781965546770981 us; error 28 us; skew 500 ppm
Jun 20 14:25:46 dist-test-slave-pk3j krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:46.773506 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:25:46.774283 1722 webserver.cc:460] Webserver started at http://127.0.98.2:43803/ using document root <none> and password file <none>
I20260620 14:25:46.774768 1722 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:46.774852 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:46.775055 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:46.776852 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "2fa13ab6b6e64d83b4992b58fa64d709"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "6a7b9e172742dc67788e49d2f23a09a2"
server_key_iv: "a3cbd96c3162579ac25f6e3107ee7930"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.777415 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2fa13ab6b6e64d83b4992b58fa64d709"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "6a7b9e172742dc67788e49d2f23a09a2"
server_key_iv: "a3cbd96c3162579ac25f6e3107ee7930"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.780553 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:25:46.782647 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:46.783630 1722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:25:46.783751 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "2fa13ab6b6e64d83b4992b58fa64d709"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "6a7b9e172742dc67788e49d2f23a09a2"
server_key_iv: "a3cbd96c3162579ac25f6e3107ee7930"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.783847 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:46.796331 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:46.799309 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:46.799481 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:46.800657 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:46.800727 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:46.800812 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:46.800860 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:46.810030 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41491
I20260620 14:25:46.810079 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41491 every 8 connection(s)
Jun 20 14:25:46 dist-test-slave-pk3j krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:46.810977 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:25:46.814003 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:46.801415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34793 (local address 127.0.98.62:46439)
0620 14:25:46.801554 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:46.801559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:46.802337 (+ 778us) server_negotiation.cc:407] Connection header received
0620 14:25:46.803363 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:46.803367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:46.803428 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:46.803521 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:46.805072 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.805992 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.807750 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.807951 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.808029 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:46.810841 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:46.810844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:46.810846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:46.810868 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:46.812390 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:46.812800 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:46.812803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:46.812804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:46.812873 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:46.813143 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:46.813145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:46.813146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:46.813351 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:46.813426 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:46.813810 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 14:25:46.813895 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:25:46.814697 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:46.801788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46439 (local address 127.0.98.2:34793)
0620 14:25:46.802190 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:46.802233 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:46.803142 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:46.803815 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:46.803824 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:46.804236 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:46.804896 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:46.804930 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.806151 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.806155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:46.807625 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:46.807636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.808096 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.808104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:46.808338 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:46.808935 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:46.808956 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:46.810671 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:46.812480 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:46.812484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:46.812494 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:46.812718 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:46.812956 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:46.812958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:46.812960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:46.813073 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:46.813437 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:46.813441 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:46.813625 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:25:46.813767 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 14:25:46.813966 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":116,"threads_started":1}
I20260620 14:25:46.814904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 1722
I20260620 14:25:46.815038 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:25:46.815282 392 external_mini_cluster.cc:1428] Setting key 4051b43d0d68f64d52a463f8d8102388
I20260620 14:25:46.816023 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46439
I20260620 14:25:46.816188 1853 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:46.816732 1853 heartbeater.cc:511] Master 127.0.98.62:46439 requested a full tablet report, sending...
I20260620 14:25:46.817752 1510 ts_manager.cc:194] Registered new tserver with Master: 2fa13ab6b6e64d83b4992b58fa64d709 (127.0.98.2:41491)
I20260620 14:25:46.818516 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34793
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:25:46.825765 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:46.818599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58247 (local address 127.0.98.62:46439)
0620 14:25:46.818690 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:46.818694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:46.818705 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:25:46.818733 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:46.818735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:46.818780 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:46.818851 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:46.819372 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.820050 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.821316 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.821509 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.821583 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:46.822795 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:46.822798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:46.822800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:46.822818 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:46.824102 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:46.824588 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:46.824590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:46.824592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:46.824654 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:46.824937 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:46.824939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:46.824940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:46.825135 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:46.825253 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:46.825562 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 14:25:46.825669 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:25:46.825786 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:46.818094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46439 (local address 127.0.98.2:58247)
0620 14:25:46.818391 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:46.818407 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:46.818503 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:46.818878 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:46.818882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:46.819093 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:46.819252 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:46.819261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.820191 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.820197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:46.821173 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:46.821186 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:46.821621 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:46.821627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:46.821746 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:46.822185 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:46.822204 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:46.822655 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:46.824209 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:46.824215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:46.824218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:46.824496 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:46.824755 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:46.824761 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:46.824764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:46.824855 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:46.825257 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:46.825261 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:46.825382 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:25:46.825498 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:25:46.825660 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":6,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:25:46.841322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46439
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:46.946555 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:46.946759 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:46.946798 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:46.950505 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:46.950546 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:46.950579 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:25:46.950641 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:25:46.955341 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:46.956210 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:46.957103 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:46.965373 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:46.965495 1862 server_base.cc:1034] running on GCE node
W20260620 14:25:46.965399 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:46.965592 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:46.966046 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:46.966670 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:46.967866 1862 hybrid_clock.cc:648] HybridClock initialized: now 1781965546967844 us; error 34 us; skew 500 ppm
Jun 20 14:25:46 dist-test-slave-pk3j krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:46.970392 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:25:46.971082 1862 webserver.cc:460] Webserver started at http://127.0.98.3:42197/ using document root <none> and password file <none>
I20260620 14:25:46.971525 1862 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:46.971577 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:46.971715 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:46.973555 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "ab1f45de58014706a14dbb00422cbaec"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "8d0cd2125c9e9c448c2c75580cd58565"
server_key_iv: "925a35425f87d1a15e129676f4847480"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.974084 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ab1f45de58014706a14dbb00422cbaec"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "8d0cd2125c9e9c448c2c75580cd58565"
server_key_iv: "925a35425f87d1a15e129676f4847480"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.977376 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:25:46.979400 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:46.980484 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:46.980569 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "ab1f45de58014706a14dbb00422cbaec"
format_stamp: "Formatted at 2026-06-20 14:25:46 on dist-test-slave-pk3j"
server_key: "8d0cd2125c9e9c448c2c75580cd58565"
server_key_iv: "925a35425f87d1a15e129676f4847480"
server_key_version: "encryptionkey@0"
I20260620 14:25:46.980640 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:46.997753 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:47.000355 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:47.000484 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:47.001760 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:47.001837 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:47.001884 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:47.001922 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:25:47 dist-test-slave-pk3j krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:47.012203 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34551
I20260620 14:25:47.012290 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34551 every 8 connection(s)
I20260620 14:25:47.013118 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:25:47.015007 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.002331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58215 (local address 127.0.98.62:46439)
0620 14:25:47.002443 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.002450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.003334 (+ 884us) server_negotiation.cc:407] Connection header received
0620 14:25:47.004185 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.004189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.004246 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.004328 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:47.005852 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.006723 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.007817 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.008040 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.008149 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.011592 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:47.011595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:47.011597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:47.011623 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:47.013157 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.013811 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.013817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.013819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.013902 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.014204 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.014206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.014208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.014362 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:47.014442 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.014813 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.014898 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:25:47.015723 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.002772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46439 (local address 127.0.98.3:58215)
0620 14:25:47.003195 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.003231 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.003971 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.004526 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.004534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.004989 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:47.005671 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.005689 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.006835 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.006840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.007694 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.007703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.008193 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.008197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.008344 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.009050 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:47.009086 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:47.011442 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:47.013313 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:47.013318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:47.013330 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:47.013680 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:47.013982 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:47.013984 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:47.013985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:47.014112 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:47.014446 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:47.014450 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:47.014634 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.014769 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.014965 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260620 14:25:47.016494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 1862
I20260620 14:25:47.016625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:25:47.016880 392 external_mini_cluster.cc:1428] Setting key a726f83876b4b66ea6065f7226ffaf4f
I20260620 14:25:47.016955 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46439
I20260620 14:25:47.017077 1993 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:47.017590 1993 heartbeater.cc:511] Master 127.0.98.62:46439 requested a full tablet report, sending...
I20260620 14:25:47.018577 1509 ts_manager.cc:194] Registered new tserver with Master: ab1f45de58014706a14dbb00422cbaec (127.0.98.3:34551)
I20260620 14:25:47.019191 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58215
I20260620 14:25:47.025640 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.019685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34663 (local address 127.0.98.62:46439)
0620 14:25:47.019785 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.019789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.019946 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:25:47.020025 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.020028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.020069 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.020145 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:47.020504 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.021194 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.021953 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.022147 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.022220 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.023041 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:47.023044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:47.023046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:47.023065 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:47.024395 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.024723 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.024725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.024726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.024787 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.024996 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.024998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.024999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.025131 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:47.025250 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.025442 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.025517 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:25:47.025672 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.019616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46439 (local address 127.0.98.3:34663)
0620 14:25:47.019866 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.019877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.019951 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.020163 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.020165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.020307 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:47.020397 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.020402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.021322 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.021324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.021859 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.021866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.022236 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.022238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.022298 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.022660 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:47.022670 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:47.022930 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:47.024481 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:47.024483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:47.024485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:47.024646 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:47.024860 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:47.024861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:47.024863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:47.024925 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:47.025247 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:47.025250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:47.025327 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.025404 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.025488 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":114,"threads_started":1}
I20260620 14:25:47.030236 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:25:47 dist-test-slave-pk3j krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:25:47 dist-test-slave-pk3j krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:47.053519 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.045796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.0.98.62:46439)
0620 14:25:47.045927 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.045930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.045940 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:25:47.045966 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.045968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.045997 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.046081 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:47.046959 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.047616 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.048419 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.048631 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.048733 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.050877 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:47.050880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:47.050883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:47.050910 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:47.051972 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.052364 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.052369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.052371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.052444 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.052707 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.052711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.052713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.052861 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:47.053023 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.053280 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.053380 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:25:47.055863 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:25:47.058785 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:25:47.071420 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.066038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43985 (local address 127.0.0.1:40760)
0620 14:25:47.066510 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.066552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.066664 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.067416 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.067422 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.067442 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:47.067556 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.067564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.068945 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.068951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.069976 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.069988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.070916 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.070920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.071015 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.071047 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.071190 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.071294 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260620 14:25:47.071439 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.066170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.0.98.1:43985)
0620 14:25:47.066959 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.066969 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.066992 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:25:47.067064 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.067069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.067261 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.067428 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:47.067702 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.068828 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.070106 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.070787 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.070913 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.071063 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.071219 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.071308 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":87,"threads_started":1}
I20260620 14:25:47.071599 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.066143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41491 (local address 127.0.0.1:38026)
0620 14:25:47.066983 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.066996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.067080 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.067773 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.067776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.067792 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:47.067914 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.067919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.069030 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.069033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.070208 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.070217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.071241 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.071245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.071338 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.071359 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.071406 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.071470 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":83,"threads_started":1}
I20260620 14:25:47.072120 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.066685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34551 (local address 127.0.0.1:40656)
0620 14:25:47.067667 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.067682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.067767 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.068164 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.068167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.068187 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:47.068256 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.068261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.069549 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.069552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.070597 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.070605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.071743 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.071749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.071831 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.071851 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.071919 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.071988 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":94,"threads_started":1}
I20260620 14:25:47.072501 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.066804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.98.3:34551)
0620 14:25:47.067304 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.067315 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.067754 (+ 439us) server_negotiation.cc:407] Connection header received
0620 14:25:47.067868 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.067873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.068028 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.068176 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:47.068371 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.069429 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.070720 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.071447 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.071531 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.071665 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.072277 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.072360 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":107,"threads_started":1}
I20260620 14:25:47.072599 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.066763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.0.98.2:41491)
0620 14:25:47.067298 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.067310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.067334 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:25:47.067402 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.067407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.067571 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.068027 (+ 456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:47.068150 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.068921 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.070338 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.071110 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.072059 (+ 949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.072288 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.072335 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.072438 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":132,"threads_started":1}
I20260620 14:25:47.074221 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6f95083fab48fa99a6acfe4181fba2 (DEFAULT_TABLE table=test-table [id=f93654322202464bad9f9d8a9a10dcdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:47.075062 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6f95083fab48fa99a6acfe4181fba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:47.074808 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6f95083fab48fa99a6acfe4181fba2 (DEFAULT_TABLE table=test-table [id=f93654322202464bad9f9d8a9a10dcdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:47.075567 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6f95083fab48fa99a6acfe4181fba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:47.081108 2016 tablet_bootstrap.cc:492] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: Bootstrap starting.
I20260620 14:25:47.081188 2017 tablet_bootstrap.cc:492] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: Bootstrap starting.
I20260620 14:25:47.082986 2017 tablet_bootstrap.cc:654] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:47.083488 2016 tablet_bootstrap.cc:654] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:47.083240 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6f95083fab48fa99a6acfe4181fba2 (DEFAULT_TABLE table=test-table [id=f93654322202464bad9f9d8a9a10dcdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:47.083909 2017 log.cc:824] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:47.084137 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6f95083fab48fa99a6acfe4181fba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:47.084249 2016 log.cc:824] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:47.086143 2016 tablet_bootstrap.cc:492] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: No bootstrap required, opened a new log
I20260620 14:25:47.086138 2017 tablet_bootstrap.cc:492] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: No bootstrap required, opened a new log
I20260620 14:25:47.086318 2016 ts_tablet_manager.cc:1397] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:25:47.086318 2017 ts_tablet_manager.cc:1397] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 14:25:47.090061 2017 raft_consensus.cc:348] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.090061 2016 raft_consensus.cc:348] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.090299 2016 raft_consensus.cc:374] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:47.090337 2017 raft_consensus.cc:374] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:47.090381 2016 raft_consensus.cc:729] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfdb84b4aee94b7faa68ff55a5405174, State: Initialized, Role: FOLLOWER
I20260620 14:25:47.090415 2017 raft_consensus.cc:729] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa13ab6b6e64d83b4992b58fa64d709, State: Initialized, Role: FOLLOWER
I20260620 14:25:47.090737 2020 tablet_bootstrap.cc:492] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: Bootstrap starting.
I20260620 14:25:47.090850 2016 consensus_queue.cc:260] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.090850 2017 consensus_queue.cc:260] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.091465 1853 heartbeater.cc:503] Master 127.0.98.62:46439 was elected leader, sending a full tablet report...
I20260620 14:25:47.091672 1713 heartbeater.cc:503] Master 127.0.98.62:46439 was elected leader, sending a full tablet report...
I20260620 14:25:47.091766 2017 ts_tablet_manager.cc:1428] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:25:47.091766 2016 ts_tablet_manager.cc:1428] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:25:47.093559 2020 tablet_bootstrap.cc:654] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:47.094188 2020 log.cc:824] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:47.096308 2020 tablet_bootstrap.cc:492] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: No bootstrap required, opened a new log
I20260620 14:25:47.096629 2020 ts_tablet_manager.cc:1397] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:25:47.099565 2020 raft_consensus.cc:348] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.099738 2020 raft_consensus.cc:374] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:47.099813 2020 raft_consensus.cc:729] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1f45de58014706a14dbb00422cbaec, State: Initialized, Role: FOLLOWER
I20260620 14:25:47.100216 2020 consensus_queue.cc:260] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.100764 1993 heartbeater.cc:503] Master 127.0.98.62:46439 was elected leader, sending a full tablet report...
I20260620 14:25:47.101094 2020 ts_tablet_manager.cc:1428] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260620 14:25:47.124082 1714 tablet.cc:2369] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:25:47.243649 2022 raft_consensus.cc:482] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:25:47.243876 2022 raft_consensus.cc:504] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.245059 2022 leader_election.cc:288] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fa13ab6b6e64d83b4992b58fa64d709 (127.0.98.2:41491), ab1f45de58014706a14dbb00422cbaec (127.0.98.3:34551)
I20260620 14:25:47.250177 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.245385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37307 (local address 127.0.98.2:41491)
0620 14:25:47.245525 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.245529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.245686 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:25:47.245854 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.245857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.245907 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.245989 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:47.246436 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.247222 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.248930 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.249539 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.249647 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.249704 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.249993 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.250059 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:25:47.250177 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.245556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48075 (local address 127.0.98.3:34551)
0620 14:25:47.245715 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.245721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.245734 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:25:47.245854 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.245857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.245906 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.245989 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:47.246436 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.247221 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.248930 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.249538 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.249647 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.249704 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.249993 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.250059 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 14:25:47.250237 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.245272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41491 (local address 127.0.98.1:37307)
0620 14:25:47.245579 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.245593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.245724 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.246023 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.246026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.246066 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:47.246328 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.246333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.247333 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.247337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.248805 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.248816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.249663 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.249667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.249809 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.249882 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.249963 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.250077 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":316,"spinlock_wait_cycles":28800,"thread_start_us":77,"threads_started":1}
I20260620 14:25:47.250240 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.245310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34551 (local address 127.0.98.1:48075)
0620 14:25:47.245675 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.245687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.245746 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.246023 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.246026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.246068 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:47.246317 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.246325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.247333 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.247337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.248812 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.248817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.249663 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.249667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.249831 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.249888 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.249969 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.250071 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":334,"spinlock_wait_cycles":7040,"thread_start_us":69,"threads_started":1}
I20260620 14:25:47.251045 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f95083fab48fa99a6acfe4181fba2" candidate_uuid: "dfdb84b4aee94b7faa68ff55a5405174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1f45de58014706a14dbb00422cbaec" is_pre_election: true
I20260620 14:25:47.251051 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f95083fab48fa99a6acfe4181fba2" candidate_uuid: "dfdb84b4aee94b7faa68ff55a5405174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" is_pre_election: true
I20260620 14:25:47.251324 1947 raft_consensus.cc:2454] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfdb84b4aee94b7faa68ff55a5405174 in term 0.
I20260620 14:25:47.251327 1807 raft_consensus.cc:2454] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfdb84b4aee94b7faa68ff55a5405174 in term 0.
I20260620 14:25:47.251762 1599 leader_election.cc:302] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1f45de58014706a14dbb00422cbaec, dfdb84b4aee94b7faa68ff55a5405174; no voters:
I20260620 14:25:47.252015 2022 raft_consensus.cc:2781] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:25:47.252086 2022 raft_consensus.cc:482] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:25:47.252140 2022 raft_consensus.cc:3037] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:47.253136 2022 raft_consensus.cc:504] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.253515 2022 leader_election.cc:288] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [CANDIDATE]: Term 1 election: Requested vote from peers 2fa13ab6b6e64d83b4992b58fa64d709 (127.0.98.2:41491), ab1f45de58014706a14dbb00422cbaec (127.0.98.3:34551)
I20260620 14:25:47.253768 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f95083fab48fa99a6acfe4181fba2" candidate_uuid: "dfdb84b4aee94b7faa68ff55a5405174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa13ab6b6e64d83b4992b58fa64d709"
I20260620 14:25:47.253809 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6f95083fab48fa99a6acfe4181fba2" candidate_uuid: "dfdb84b4aee94b7faa68ff55a5405174" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1f45de58014706a14dbb00422cbaec"
I20260620 14:25:47.253886 1807 raft_consensus.cc:3037] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:47.253919 1947 raft_consensus.cc:3037] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:47.254880 1947 raft_consensus.cc:2454] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfdb84b4aee94b7faa68ff55a5405174 in term 1.
I20260620 14:25:47.255162 1599 leader_election.cc:302] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1f45de58014706a14dbb00422cbaec, dfdb84b4aee94b7faa68ff55a5405174; no voters:
I20260620 14:25:47.255286 1807 raft_consensus.cc:2454] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfdb84b4aee94b7faa68ff55a5405174 in term 1.
I20260620 14:25:47.255357 2022 raft_consensus.cc:2781] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:47.255563 2022 raft_consensus.cc:686] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 LEADER]: Becoming Leader. State: Replica: dfdb84b4aee94b7faa68ff55a5405174, State: Running, Role: LEADER
I20260620 14:25:47.255889 2022 consensus_queue.cc:237] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.258668 1509 catalog_manager.cc:5495] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfdb84b4aee94b7faa68ff55a5405174 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dfdb84b4aee94b7faa68ff55a5405174" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } health_report { overall_health: UNKNOWN } } }
W20260620 14:25:47.264170 1994 tablet.cc:2369] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:25:47.308727 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.305273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.0.98.1:43985)
0620 14:25:47.305407 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.305411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.305424 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:25:47.305640 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.305644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.305701 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.305794 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:47.306049 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.306532 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.307394 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.307604 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.307710 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.308013 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:47.308359 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:47.308477 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.308550 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.308599 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 14:25:47.312088 1854 tablet.cc:2369] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:25:47.314131 1807 raft_consensus.cc:1261] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 FOLLOWER]: Refusing update from remote peer dfdb84b4aee94b7faa68ff55a5405174: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:47.314139 1947 raft_consensus.cc:1261] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec [term 1 FOLLOWER]: Refusing update from remote peer dfdb84b4aee94b7faa68ff55a5405174: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:47.314710 2027 consensus_queue.cc:1035] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:47.314869 2022 consensus_queue.cc:1035] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:47.324069 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298930881792696320 to 7298930881562705920. Current Snapshot: MvccSnapshot[applied={T|T < 7298930881792696320}]
I20260620 14:25:47.332350 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:47.332474 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:47.334451 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b: Sending DeleteTablet for 3 replicas of tablet 2f6f95083fab48fa99a6acfe4181fba2
I20260620 14:25:47.335143 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6f95083fab48fa99a6acfe4181fba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40656
I20260620 14:25:47.335121 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6f95083fab48fa99a6acfe4181fba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38026
I20260620 14:25:47.335259 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6f95083fab48fa99a6acfe4181fba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40760
I20260620 14:25:47.335624 2040 tablet_replica.cc:330] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: stopping tablet replica
I20260620 14:25:47.335664 2042 tablet_replica.cc:330] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: stopping tablet replica
I20260620 14:25:47.335948 2040 raft_consensus.cc:2229] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:47.335953 2042 raft_consensus.cc:2229] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:25:47.336198 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:25:47.336282 2040 raft_consensus.cc:2258] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:47.336295 2042 raft_consensus.cc:2258] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:25:47.336560 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:25:47.336925 2041 tablet_replica.cc:330] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: stopping tablet replica
I20260620 14:25:47.337226 2041 raft_consensus.cc:2229] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:47.337507 2041 raft_consensus.cc:2258] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:47.337929 2042 ts_tablet_manager.cc:1905] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:47.338498 2040 ts_tablet_manager.cc:1905] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:47.340546 2042 ts_tablet_manager.cc:1918] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:25:47.340691 2042 log.cc:1196] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/2f6f95083fab48fa99a6acfe4181fba2
I20260620 14:25:47.340945 2040 ts_tablet_manager.cc:1918] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:25:47.341066 2040 log.cc:1196] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/2f6f95083fab48fa99a6acfe4181fba2
I20260620 14:25:47.341109 2042 ts_tablet_manager.cc:1939] T 2f6f95083fab48fa99a6acfe4181fba2 P dfdb84b4aee94b7faa68ff55a5405174: Deleting consensus metadata
I20260620 14:25:47.341424 2040 ts_tablet_manager.cc:1939] T 2f6f95083fab48fa99a6acfe4181fba2 P 2fa13ab6b6e64d83b4992b58fa64d709: Deleting consensus metadata
I20260620 14:25:47.342039 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 6544bbe0e8cf4f91927eeefaf6ecd055 (DEFAULT_TABLE table=test-table [id=aa5eefb809c84b62841777377964cd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:47.342269 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6544bbe0e8cf4f91927eeefaf6ecd055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:47.342475 1494 catalog_manager.cc:4841] TS 2fa13ab6b6e64d83b4992b58fa64d709 (127.0.98.2:41491): tablet 2f6f95083fab48fa99a6acfe4181fba2 (table test-table [id=f93654322202464bad9f9d8a9a10dcdc]) successfully deleted
I20260620 14:25:47.343389 2041 ts_tablet_manager.cc:1905] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:47.343889 2016 tablet_bootstrap.cc:492] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174: Bootstrap starting.
I20260620 14:25:47.344096 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 6544bbe0e8cf4f91927eeefaf6ecd055 (DEFAULT_TABLE table=test-table [id=aa5eefb809c84b62841777377964cd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:47.344321 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6544bbe0e8cf4f91927eeefaf6ecd055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:47.344801 2016 tablet_bootstrap.cc:654] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:47.345875 2041 ts_tablet_manager.cc:1918] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:25:47.345901 2016 tablet_bootstrap.cc:492] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174: No bootstrap required, opened a new log
I20260620 14:25:47.345949 2041 log.cc:1196] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/2f6f95083fab48fa99a6acfe4181fba2
I20260620 14:25:47.345970 2016 ts_tablet_manager.cc:1397] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 14:25:47.346246 2041 ts_tablet_manager.cc:1939] T 2f6f95083fab48fa99a6acfe4181fba2 P ab1f45de58014706a14dbb00422cbaec: Deleting consensus metadata
I20260620 14:25:47.346395 2016 raft_consensus.cc:348] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.346503 2016 raft_consensus.cc:374] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:47.346537 2016 raft_consensus.cc:729] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfdb84b4aee94b7faa68ff55a5405174, State: Initialized, Role: FOLLOWER
I20260620 14:25:47.346575 2020 tablet_bootstrap.cc:492] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec: Bootstrap starting.
I20260620 14:25:47.346661 2016 consensus_queue.cc:260] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.346921 2016 ts_tablet_manager.cc:1428] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:25:47.347390 2020 tablet_bootstrap.cc:654] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:47.347926 1495 catalog_manager.cc:4841] TS dfdb84b4aee94b7faa68ff55a5405174 (127.0.98.1:43985): tablet 2f6f95083fab48fa99a6acfe4181fba2 (table test-table [id=f93654322202464bad9f9d8a9a10dcdc]) successfully deleted
I20260620 14:25:47.348357 2020 tablet_bootstrap.cc:492] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec: No bootstrap required, opened a new log
I20260620 14:25:47.348344 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 6544bbe0e8cf4f91927eeefaf6ecd055 (DEFAULT_TABLE table=test-table [id=aa5eefb809c84b62841777377964cd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:47.348480 2020 ts_tablet_manager.cc:1397] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:25:47.348564 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6544bbe0e8cf4f91927eeefaf6ecd055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:47.348832 2020 raft_consensus.cc:348] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.348940 2020 raft_consensus.cc:374] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:47.348978 2020 raft_consensus.cc:729] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1f45de58014706a14dbb00422cbaec, State: Initialized, Role: FOLLOWER
I20260620 14:25:47.349102 2020 consensus_queue.cc:260] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.349964 2020 ts_tablet_manager.cc:1428] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:47.350555 2017 tablet_bootstrap.cc:492] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709: Bootstrap starting.
I20260620 14:25:47.351359 2017 tablet_bootstrap.cc:654] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:47.351874 1495 catalog_manager.cc:4841] TS ab1f45de58014706a14dbb00422cbaec (127.0.98.3:34551): tablet 2f6f95083fab48fa99a6acfe4181fba2 (table test-table [id=f93654322202464bad9f9d8a9a10dcdc]) successfully deleted
I20260620 14:25:47.352573 2017 tablet_bootstrap.cc:492] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709: No bootstrap required, opened a new log
I20260620 14:25:47.352656 2017 ts_tablet_manager.cc:1397] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 14:25:47.353042 2017 raft_consensus.cc:348] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.353132 2017 raft_consensus.cc:374] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:47.353165 2017 raft_consensus.cc:729] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa13ab6b6e64d83b4992b58fa64d709, State: Initialized, Role: FOLLOWER
I20260620 14:25:47.353302 2017 consensus_queue.cc:260] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.353950 2017 ts_tablet_manager.cc:1428] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:25:47.372483 2021 raft_consensus.cc:482] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:25:47.372586 2021 raft_consensus.cc:504] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.373548 2021 leader_election.cc:288] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfdb84b4aee94b7faa68ff55a5405174 (127.0.98.1:43985), ab1f45de58014706a14dbb00422cbaec (127.0.98.3:34551)
I20260620 14:25:47.378710 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.373795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49861 (local address 127.0.98.1:43985)
0620 14:25:47.373904 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.373909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.373990 (+ 81us) server_negotiation.cc:407] Connection header received
0620 14:25:47.374187 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.374190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.374243 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.374287 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:47.374938 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.375649 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.377461 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.378029 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.378094 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.378146 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.378539 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.378604 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:25:47.378710 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.374106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48827 (local address 127.0.98.3:34551)
0620 14:25:47.374192 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.374196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.374376 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:25:47.374533 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.374535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.374573 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.374618 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:47.374942 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.375650 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.377463 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.378025 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.378094 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.378145 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.378539 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.378604 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:25:47.379151 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.374019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34551 (local address 127.0.98.2:48827)
0620 14:25:47.374315 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.374327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.374436 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.374728 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.374730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.374744 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:47.374836 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.374843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.375786 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.375790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.377331 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.377340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.378245 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.378249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.378387 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.378412 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.378898 (+ 486us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.378983 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":194,"thread_start_us":68,"threads_started":1}
I20260620 14:25:47.379321 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.373684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43985 (local address 127.0.98.2:49861)
0620 14:25:47.373908 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.373924 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.374057 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.374422 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.374424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.374443 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:47.374811 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.374819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.375786 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.375790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.377331 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.377340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.378246 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.378249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.378391 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.378409 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.379118 (+ 709us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.379187 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":146,"spinlock_wait_cycles":768,"thread_start_us":87,"threads_started":1}
I20260620 14:25:47.379647 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6544bbe0e8cf4f91927eeefaf6ecd055" candidate_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1f45de58014706a14dbb00422cbaec" is_pre_election: true
I20260620 14:25:47.379769 1947 raft_consensus.cc:2454] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa13ab6b6e64d83b4992b58fa64d709 in term 0.
I20260620 14:25:47.379951 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6544bbe0e8cf4f91927eeefaf6ecd055" candidate_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfdb84b4aee94b7faa68ff55a5405174" is_pre_election: true
I20260620 14:25:47.380172 1667 raft_consensus.cc:2454] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fa13ab6b6e64d83b4992b58fa64d709 in term 0.
I20260620 14:25:47.380561 1739 leader_election.cc:302] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa13ab6b6e64d83b4992b58fa64d709, ab1f45de58014706a14dbb00422cbaec; no voters:
I20260620 14:25:47.380759 2021 raft_consensus.cc:2781] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:25:47.380805 2021 raft_consensus.cc:482] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:25:47.380838 2021 raft_consensus.cc:3037] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:47.381783 2021 raft_consensus.cc:504] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.382042 2021 leader_election.cc:288] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [CANDIDATE]: Term 1 election: Requested vote from peers dfdb84b4aee94b7faa68ff55a5405174 (127.0.98.1:43985), ab1f45de58014706a14dbb00422cbaec (127.0.98.3:34551)
I20260620 14:25:47.382349 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6544bbe0e8cf4f91927eeefaf6ecd055" candidate_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1f45de58014706a14dbb00422cbaec"
I20260620 14:25:47.382460 1947 raft_consensus.cc:3037] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:47.382481 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6544bbe0e8cf4f91927eeefaf6ecd055" candidate_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfdb84b4aee94b7faa68ff55a5405174"
I20260620 14:25:47.382570 1667 raft_consensus.cc:3037] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:47.383448 1947 raft_consensus.cc:2454] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa13ab6b6e64d83b4992b58fa64d709 in term 1.
I20260620 14:25:47.383648 1667 raft_consensus.cc:2454] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fa13ab6b6e64d83b4992b58fa64d709 in term 1.
I20260620 14:25:47.383912 1739 leader_election.cc:302] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa13ab6b6e64d83b4992b58fa64d709, ab1f45de58014706a14dbb00422cbaec; no voters:
I20260620 14:25:47.384161 2021 raft_consensus.cc:2781] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:47.384379 2021 raft_consensus.cc:686] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa13ab6b6e64d83b4992b58fa64d709, State: Running, Role: LEADER
I20260620 14:25:47.384580 2021 consensus_queue.cc:237] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.386538 1510 catalog_manager.cc:5495] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fa13ab6b6e64d83b4992b58fa64d709 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:25:47 dist-test-slave-pk3j krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:47.427523 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.418655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46439 (local address 127.0.0.1:37818)
0620 14:25:47.418962 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.418977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.419057 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.419291 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.419293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.419617 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:47.419692 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.419697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.420722 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.420724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.421308 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.421321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.421875 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.421877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.421933 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.422534 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:47.422546 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:47.424020 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:47.425800 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:47.425808 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:47.425812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:47.426025 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:47.426575 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:47.426578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:47.426580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:47.426655 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:47.427026 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:47.427034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:47.427129 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.427250 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.427360 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":100,"threads_started":1}
I20260620 14:25:47.427610 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.418704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.0.98.62:46439)
0620 14:25:47.418821 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.418825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.419046 (+ 221us) server_negotiation.cc:407] Connection header received
0620 14:25:47.419135 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.419138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.419179 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.419262 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:47.419812 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.420611 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.421430 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.421752 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.421849 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.424154 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:47.424157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:47.424160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:47.424183 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:47.425702 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.426388 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.426393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.426395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.426474 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.426742 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.426747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.426749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.426920 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:47.427018 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.427316 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.427451 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:25:47.431619 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37818:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:25:47.432369 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:25:47.437152 1927 tablet_service.cc:1467] Processing CreateTablet for tablet e3adfc0a0ef440c78c067c1e1ee01fd9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b19692497d84bb98381bf73dc4e5295]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:25:47.437583 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3adfc0a0ef440c78c067c1e1ee01fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:47.437791 1787 tablet_service.cc:1467] Processing CreateTablet for tablet e3adfc0a0ef440c78c067c1e1ee01fd9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b19692497d84bb98381bf73dc4e5295]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:25:47.438122 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3adfc0a0ef440c78c067c1e1ee01fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:47.439713 2020 tablet_bootstrap.cc:492] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: Bootstrap starting.
I20260620 14:25:47.436956 1647 tablet_service.cc:1467] Processing CreateTablet for tablet e3adfc0a0ef440c78c067c1e1ee01fd9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8b19692497d84bb98381bf73dc4e5295]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:25:47.440320 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3adfc0a0ef440c78c067c1e1ee01fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:47.440621 2020 tablet_bootstrap.cc:654] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:47.442051 2017 tablet_bootstrap.cc:492] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: Bootstrap starting.
I20260620 14:25:47.442636 2020 tablet_bootstrap.cc:492] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: No bootstrap required, opened a new log
I20260620 14:25:47.442703 2020 ts_tablet_manager.cc:1397] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:25:47.443002 2017 tablet_bootstrap.cc:654] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:47.443145 2020 raft_consensus.cc:348] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.443264 2020 raft_consensus.cc:374] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:47.443298 2020 raft_consensus.cc:729] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1f45de58014706a14dbb00422cbaec, State: Initialized, Role: FOLLOWER
I20260620 14:25:47.443439 2020 consensus_queue.cc:260] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.443711 2020 ts_tablet_manager.cc:1428] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:47.443933 2016 tablet_bootstrap.cc:492] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: Bootstrap starting.
I20260620 14:25:47.444794 2016 tablet_bootstrap.cc:654] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:47.445842 2024 tablet_replica.cc:439] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } } }
I20260620 14:25:47.445998 2024 tablet_replica.cc:442] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:47.446486 2016 tablet_bootstrap.cc:492] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: No bootstrap required, opened a new log
I20260620 14:25:47.446586 2016 ts_tablet_manager.cc:1397] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:25:47.446969 2016 raft_consensus.cc:348] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.447073 2016 raft_consensus.cc:374] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:47.447115 2016 raft_consensus.cc:729] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfdb84b4aee94b7faa68ff55a5405174, State: Initialized, Role: FOLLOWER
I20260620 14:25:47.447256 2016 consensus_queue.cc:260] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.447480 2035 tablet_replica.cc:439] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } } }
I20260620 14:25:47.447593 2035 tablet_replica.cc:442] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:47.448012 2016 ts_tablet_manager.cc:1428] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:47.449056 2017 tablet_bootstrap.cc:492] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: No bootstrap required, opened a new log
I20260620 14:25:47.449156 2017 ts_tablet_manager.cc:1397] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260620 14:25:47.449638 2017 raft_consensus.cc:348] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.449771 2017 raft_consensus.cc:374] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:47.449805 2017 raft_consensus.cc:729] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa13ab6b6e64d83b4992b58fa64d709, State: Initialized, Role: FOLLOWER
I20260620 14:25:47.449930 2017 consensus_queue.cc:260] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.450191 2017 ts_tablet_manager.cc:1428] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:25:47.450212 2021 tablet_replica.cc:439] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } } }
I20260620 14:25:47.450316 2021 tablet_replica.cc:442] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:47.594229 2024 raft_consensus.cc:482] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:25:47.594339 2024 raft_consensus.cc:504] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.595424 2024 leader_election.cc:288] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfdb84b4aee94b7faa68ff55a5405174 (127.0.98.1:43985), 2fa13ab6b6e64d83b4992b58fa64d709 (127.0.98.2:41491)
I20260620 14:25:47.600569 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.595809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44161 (local address 127.0.98.2:41491)
0620 14:25:47.595935 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.595938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.596145 (+ 207us) server_negotiation.cc:407] Connection header received
0620 14:25:47.596246 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.596249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.596295 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.596377 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:47.596878 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.597630 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.599368 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.599929 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.600043 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.600102 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.600384 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.600446 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:25:47.600569 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.595678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32987 (local address 127.0.98.1:43985)
0620 14:25:47.595808 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.595815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.596064 (+ 249us) server_negotiation.cc:407] Connection header received
0620 14:25:47.596219 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.596224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.596274 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.596360 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:47.596819 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.597605 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.599384 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.599959 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.600048 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.600102 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.600382 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.600446 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:25:47.600658 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.595657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41491 (local address 127.0.98.3:44161)
0620 14:25:47.596060 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.596072 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.596151 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.596391 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.596394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.596433 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:47.596718 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.596724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.597726 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.597745 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.599236 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.599247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.600063 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.600066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.600229 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.600279 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.600352 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.600467 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":295,"spinlock_wait_cycles":29056,"thread_start_us":83,"threads_started":1}
I20260620 14:25:47.600658 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.595578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43985 (local address 127.0.98.3:32987)
0620 14:25:47.595965 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.595981 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.596100 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.596391 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.596394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.596434 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:47.596703 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.596710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.597726 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.597730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.599241 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.599247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.600063 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.600073 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.600235 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.600275 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.600352 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.600467 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":247,"spinlock_wait_cycles":5888,"thread_start_us":78,"threads_started":1}
I20260620 14:25:47.601104 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3adfc0a0ef440c78c067c1e1ee01fd9" candidate_uuid: "ab1f45de58014706a14dbb00422cbaec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfdb84b4aee94b7faa68ff55a5405174" is_pre_election: true
I20260620 14:25:47.601269 1667 raft_consensus.cc:2454] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1f45de58014706a14dbb00422cbaec in term 0.
I20260620 14:25:47.601281 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3adfc0a0ef440c78c067c1e1ee01fd9" candidate_uuid: "ab1f45de58014706a14dbb00422cbaec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" is_pre_election: true
I20260620 14:25:47.601408 1807 raft_consensus.cc:2454] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab1f45de58014706a14dbb00422cbaec in term 0.
I20260620 14:25:47.601598 1879 leader_election.cc:302] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1f45de58014706a14dbb00422cbaec, dfdb84b4aee94b7faa68ff55a5405174; no voters:
I20260620 14:25:47.601838 2024 raft_consensus.cc:2781] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:25:47.601913 2024 raft_consensus.cc:482] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:25:47.601964 2024 raft_consensus.cc:3037] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:47.602880 2024 raft_consensus.cc:504] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.603189 2024 leader_election.cc:288] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [CANDIDATE]: Term 1 election: Requested vote from peers dfdb84b4aee94b7faa68ff55a5405174 (127.0.98.1:43985), 2fa13ab6b6e64d83b4992b58fa64d709 (127.0.98.2:41491)
I20260620 14:25:47.603511 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3adfc0a0ef440c78c067c1e1ee01fd9" candidate_uuid: "ab1f45de58014706a14dbb00422cbaec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa13ab6b6e64d83b4992b58fa64d709"
I20260620 14:25:47.603506 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3adfc0a0ef440c78c067c1e1ee01fd9" candidate_uuid: "ab1f45de58014706a14dbb00422cbaec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfdb84b4aee94b7faa68ff55a5405174"
I20260620 14:25:47.603616 1807 raft_consensus.cc:3037] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:47.603616 1667 raft_consensus.cc:3037] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:47.604637 1807 raft_consensus.cc:2454] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1f45de58014706a14dbb00422cbaec in term 1.
I20260620 14:25:47.604637 1667 raft_consensus.cc:2454] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab1f45de58014706a14dbb00422cbaec in term 1.
I20260620 14:25:47.604888 1879 leader_election.cc:302] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab1f45de58014706a14dbb00422cbaec, dfdb84b4aee94b7faa68ff55a5405174; no voters:
I20260620 14:25:47.605037 2024 raft_consensus.cc:2781] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:47.605233 2024 raft_consensus.cc:686] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [term 1 LEADER]: Becoming Leader. State: Replica: ab1f45de58014706a14dbb00422cbaec, State: Running, Role: LEADER
I20260620 14:25:47.605455 2024 consensus_queue.cc:237] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } }
I20260620 14:25:47.606638 2061 tablet_replica.cc:439] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: TxnStatusTablet state changed. Reason: New leader ab1f45de58014706a14dbb00422cbaec. Latest consensus state: current_term: 1 leader_uuid: "ab1f45de58014706a14dbb00422cbaec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } } }
I20260620 14:25:47.606750 2061 tablet_replica.cc:442] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:25:47.607015 2063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:25:47.607160 2063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:25:47.607779 1510 catalog_manager.cc:5495] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec reported cstate change: term changed from 0 to 1, leader changed from <none> to ab1f45de58014706a14dbb00422cbaec (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ab1f45de58014706a14dbb00422cbaec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } health_report { overall_health: HEALTHY } } }
I20260620 14:25:47.637661 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.634363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.0.98.3:34551)
0620 14:25:47.634461 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.634466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.634490 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:25:47.634549 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.634552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.634605 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.634702 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:47.634938 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.635400 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.636239 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.636455 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.636561 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.636844 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:47.637186 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:47.637326 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.637479 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.637540 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:25:47.637670 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.634165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34551 (local address 127.0.0.1:40670)
0620 14:25:47.634310 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.634324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.634431 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.634716 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.634719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.634727 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:25:47.634821 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.634833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.635507 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.635510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.636123 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.636130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.636563 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.636567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.636656 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.636698 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:25:47.637359 (+ 661us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:25:47.637370 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.637458 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.637560 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:25:47.640012 1807 raft_consensus.cc:1261] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 FOLLOWER]: Refusing update from remote peer ab1f45de58014706a14dbb00422cbaec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:47.640372 1667 raft_consensus.cc:1261] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 FOLLOWER]: Refusing update from remote peer ab1f45de58014706a14dbb00422cbaec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:47.640889 2024 consensus_queue.cc:1035] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:47.641014 2061 consensus_queue.cc:1035] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:47.642293 2021 tablet_replica.cc:439] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: TxnStatusTablet state changed. Reason: New leader ab1f45de58014706a14dbb00422cbaec. Latest consensus state: current_term: 1 leader_uuid: "ab1f45de58014706a14dbb00422cbaec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } } }
I20260620 14:25:47.642396 2021 tablet_replica.cc:442] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:47.642516 2035 tablet_replica.cc:439] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: TxnStatusTablet state changed. Reason: New leader ab1f45de58014706a14dbb00422cbaec. Latest consensus state: current_term: 1 leader_uuid: "ab1f45de58014706a14dbb00422cbaec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } } }
I20260620 14:25:47.642607 2035 tablet_replica.cc:442] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:47.644678 2061 tablet_replica.cc:439] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab1f45de58014706a14dbb00422cbaec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } } }
I20260620 14:25:47.644701 2024 tablet_replica.cc:439] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab1f45de58014706a14dbb00422cbaec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } } }
I20260620 14:25:47.644783 2061 tablet_replica.cc:442] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:25:47.644804 2024 tablet_replica.cc:442] T e3adfc0a0ef440c78c067c1e1ee01fd9 P ab1f45de58014706a14dbb00422cbaec: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:25:47.645363 2021 tablet_replica.cc:439] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab1f45de58014706a14dbb00422cbaec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } } }
I20260620 14:25:47.645466 2021 tablet_replica.cc:442] T e3adfc0a0ef440c78c067c1e1ee01fd9 P 2fa13ab6b6e64d83b4992b58fa64d709: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:47.646497 2035 tablet_replica.cc:439] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab1f45de58014706a14dbb00422cbaec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 } } peers { permanent_uuid: "2fa13ab6b6e64d83b4992b58fa64d709" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41491 } } peers { permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 } } }
I20260620 14:25:47.646597 2035 tablet_replica.cc:442] T e3adfc0a0ef440c78c067c1e1ee01fd9 P dfdb84b4aee94b7faa68ff55a5405174: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:47.647089 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298930883129303040 to 7298930882994466816. Current Snapshot: MvccSnapshot[applied={T|T < 7298930883129303040 or (T in {7298930883129303040})}]
I20260620 14:25:47.647074 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298930883129303040 to 7298930882994466816. Current Snapshot: MvccSnapshot[applied={T|T < 7298930883129303040 or (T in {7298930883129303040})}]
I20260620 14:25:47.652495 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.649367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.0.98.2:41491)
0620 14:25:47.649467 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.649471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.649536 (+ 65us) server_negotiation.cc:407] Connection header received
0620 14:25:47.649697 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.649700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.649754 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.649847 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:47.650068 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.650503 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.651259 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.651482 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.651586 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.651827 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:47.652166 (+ 339us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:47.652292 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.652346 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.652389 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:25:47.662217 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.659209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51517 (local address 127.0.98.3:34551)
0620 14:25:47.659323 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.659328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.659508 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:25:47.659622 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.659625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.659668 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.659742 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:47.659973 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.660424 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.661283 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.661484 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.661587 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.661850 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:47.661908 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:47.661984 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.662067 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.662104 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:25:47.662513 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.659123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34551 (local address 127.0.98.2:51517)
0620 14:25:47.659426 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.659437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.659543 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.659751 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.659754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.659760 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:25:47.659861 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.659876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.660526 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.660532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.661130 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.661138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.661587 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.661590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.661678 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.661721 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:25:47.661988 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:25:47.661996 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.662058 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.662125 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260620 14:25:47.668076 1667 raft_consensus.cc:1261] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 FOLLOWER]: Refusing update from remote peer 2fa13ab6b6e64d83b4992b58fa64d709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:47.668084 1947 raft_consensus.cc:1261] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec [term 1 FOLLOWER]: Refusing update from remote peer 2fa13ab6b6e64d83b4992b58fa64d709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:47.668531 2048 consensus_queue.cc:1035] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1f45de58014706a14dbb00422cbaec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:47.668716 2021 consensus_queue.cc:1035] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfdb84b4aee94b7faa68ff55a5405174" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:47.688959 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.685057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48665 (local address 127.0.98.2:41491)
0620 14:25:47.685149 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.685155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.685169 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:25:47.685317 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.685321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.685358 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.685452 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:47.685943 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.686727 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.687762 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.688096 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.688201 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.688620 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:47.688678 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:47.688738 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.688841 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.688878 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:25:47.689039 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.684750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41491 (local address 127.0.98.3:48665)
0620 14:25:47.685110 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:47.685125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:47.685234 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:47.685461 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:47.685690 (+ 229us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:47.685705 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:25:47.685818 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.685825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.686839 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.686841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.687646 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:47.687655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.688314 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.688318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:47.688407 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.688467 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:25:47.688760 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:25:47.688766 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:25:47.688834 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:25:47.688900 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":89,"threads_started":1}
I20260620 14:25:47.712422 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:47.712563 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:47.714978 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4b36902d2c945e6a5114fd0b3f2d94b: Sending DeleteTablet for 3 replicas of tablet 6544bbe0e8cf4f91927eeefaf6ecd055
I20260620 14:25:47.715631 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 6544bbe0e8cf4f91927eeefaf6ecd055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40760
I20260620 14:25:47.715677 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 6544bbe0e8cf4f91927eeefaf6ecd055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40656
I20260620 14:25:47.715813 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 6544bbe0e8cf4f91927eeefaf6ecd055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38026
I20260620 14:25:47.715981 2042 tablet_replica.cc:330] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174: stopping tablet replica
I20260620 14:25:47.716099 2042 raft_consensus.cc:2229] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:47.716207 2042 raft_consensus.cc:2258] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:47.716355 2041 tablet_replica.cc:330] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec: stopping tablet replica
I20260620 14:25:47.716462 2041 raft_consensus.cc:2229] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:47.716555 2041 raft_consensus.cc:2258] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:47.716718 2040 tablet_replica.cc:330] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709: stopping tablet replica
I20260620 14:25:47.716836 2040 raft_consensus.cc:2229] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:25:47.717095 2040 raft_consensus.cc:2258] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:47.717872 2041 ts_tablet_manager.cc:1905] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:47.718120 2042 ts_tablet_manager.cc:1905] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:47.720454 2041 ts_tablet_manager.cc:1918] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:25:47.720515 2041 log.cc:1196] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/6544bbe0e8cf4f91927eeefaf6ecd055
I20260620 14:25:47.720741 2041 ts_tablet_manager.cc:1939] T 6544bbe0e8cf4f91927eeefaf6ecd055 P ab1f45de58014706a14dbb00422cbaec: Deleting consensus metadata
I20260620 14:25:47.720757 2042 ts_tablet_manager.cc:1918] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:25:47.720819 2042 log.cc:1196] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/6544bbe0e8cf4f91927eeefaf6ecd055
I20260620 14:25:47.721060 2042 ts_tablet_manager.cc:1939] T 6544bbe0e8cf4f91927eeefaf6ecd055 P dfdb84b4aee94b7faa68ff55a5405174: Deleting consensus metadata
I20260620 14:25:47.721477 1495 catalog_manager.cc:4841] TS ab1f45de58014706a14dbb00422cbaec (127.0.98.3:34551): tablet 6544bbe0e8cf4f91927eeefaf6ecd055 (table test-table [id=aa5eefb809c84b62841777377964cd5e]) successfully deleted
I20260620 14:25:47.721740 2040 ts_tablet_manager.cc:1905] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:47.722623 1495 catalog_manager.cc:4841] TS dfdb84b4aee94b7faa68ff55a5405174 (127.0.98.1:43985): tablet 6544bbe0e8cf4f91927eeefaf6ecd055 (table test-table [id=aa5eefb809c84b62841777377964cd5e]) successfully deleted
I20260620 14:25:47.724443 2040 ts_tablet_manager.cc:1918] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:25:47.724509 2040 log.cc:1196] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/6544bbe0e8cf4f91927eeefaf6ecd055
I20260620 14:25:47.724727 2040 ts_tablet_manager.cc:1939] T 6544bbe0e8cf4f91927eeefaf6ecd055 P 2fa13ab6b6e64d83b4992b58fa64d709: Deleting consensus metadata
I20260620 14:25:47.725265 1494 catalog_manager.cc:4841] TS 2fa13ab6b6e64d83b4992b58fa64d709 (127.0.98.2:41491): tablet 6544bbe0e8cf4f91927eeefaf6ecd055 (table test-table [id=aa5eefb809c84b62841777377964cd5e]) successfully deleted
Jun 20 14:25:47 dist-test-slave-pk3j krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965547, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:25:47.732913 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.722435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.0.98.1:43985)
0620 14:25:47.722533 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.722537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.723066 (+ 529us) server_negotiation.cc:407] Connection header received
0620 14:25:47.723177 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.723183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.723236 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.723312 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:47.724192 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.724945 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.726056 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.726293 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.726403 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.729492 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:47.729501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:47.729514 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:47.729545 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:47.731230 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.731660 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.731666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.731668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.731752 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.732032 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.732035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.732037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.732265 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:47.732465 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.732615 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.732722 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 14:25:47.733273 1706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40776
I20260620 14:25:47.741782 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:47.734883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.0.98.62:46439)
0620 14:25:47.734975 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:47.734979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:47.735282 (+ 303us) server_negotiation.cc:407] Connection header received
0620 14:25:47.735405 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:47.735408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:47.735448 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:47.735534 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:47.736120 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.736619 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.737431 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:47.737656 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:47.737760 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:47.738941 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:47.738944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:47.738947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:47.738969 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:47.740329 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.740693 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.740698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.740699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.740768 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:47.741013 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:47.741017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:47.741018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:47.741167 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:47.741335 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:47.741547 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 14:25:47.741644 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 14:25:47.742231 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37828
I20260620 14:25:47.743383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 1582
I20260620 14:25:47.751973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 1722
I20260620 14:25:47.758777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 1862
I20260620 14:25:47.763819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 1478
2026-06-20T14:25:47Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5298 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:25:47 dist-test-slave-pk3j krb5kdc[2085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:25:47 dist-test-slave-pk3j krb5kdc[2085](info): set up 2 sockets
Jun 20 14:25:47 dist-test-slave-pk3j krb5kdc[2085](info): commencing operation
krb5kdc: starting...
W20260620 14:25:48.772944 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:25:48 dist-test-slave-pk3j krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:25:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:25:48.879928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43449
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:48.980399 2098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:48.980602 2098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:48.980638 2098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:48.984440 2098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:48.984484 2098 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:25:48.984500 2098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:48.984517 2098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:48.984575 2098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:48.984596 2098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:48.989395 2098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43449
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:48.990249 2098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:48.991132 2098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:48.998801 2103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:48.998808 2106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:48.998811 2104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:48.998893 2098 server_base.cc:1034] running on GCE node
I20260620 14:25:48.999480 2098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:49.000398 2098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:49.001551 2098 hybrid_clock.cc:648] HybridClock initialized: now 1781965549001509 us; error 48 us; skew 500 ppm
Jun 20 14:25:49 dist-test-slave-pk3j krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:49.009604 2098 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:25:49.010366 2098 webserver.cc:460] Webserver started at http://127.0.98.62:45211/ using document root <none> and password file <none>
I20260620 14:25:49.010859 2098 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:49.010944 2098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:49.011152 2098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:49.012949 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "b870f916dd5645ff9eca5bffe23c388d"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "b473457b0396f08a39d01d240ca30539"
server_key_iv: "c94e4cbe2017d28cebf33327dcb13d50"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.013504 2098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "b870f916dd5645ff9eca5bffe23c388d"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "b473457b0396f08a39d01d240ca30539"
server_key_iv: "c94e4cbe2017d28cebf33327dcb13d50"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.016762 2098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:25:49.018901 2113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:49.019966 2098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:25:49.020097 2098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "b870f916dd5645ff9eca5bffe23c388d"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "b473457b0396f08a39d01d240ca30539"
server_key_iv: "c94e4cbe2017d28cebf33327dcb13d50"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.020197 2098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:49.055647 2098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:49.058303 2098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:49.058494 2098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:49.065562 2098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43449
I20260620 14:25:49.065598 2175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43449 every 8 connection(s)
I20260620 14:25:49.066395 2098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:49.068843 2176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:49.073865 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d: Bootstrap starting.
I20260620 14:25:49.075155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 2098
I20260620 14:25:49.075270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:49.075523 392 external_mini_cluster.cc:1428] Setting key 9e596f5129bcdaa013fa370e26892f13
I20260620 14:25:49.076454 2176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:49.077281 2176 log.cc:824] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:49.079006 2176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d: No bootstrap required, opened a new log
I20260620 14:25:49.083096 2176 raft_consensus.cc:348] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b870f916dd5645ff9eca5bffe23c388d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } }
I20260620 14:25:49.083338 2176 raft_consensus.cc:374] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:49.083415 2176 raft_consensus.cc:729] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b870f916dd5645ff9eca5bffe23c388d, State: Initialized, Role: FOLLOWER
I20260620 14:25:49.083875 2176 consensus_queue.cc:260] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b870f916dd5645ff9eca5bffe23c388d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } }
I20260620 14:25:49.084035 2176 raft_consensus.cc:388] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:25:49.084095 2176 raft_consensus.cc:482] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:25:49.084201 2176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:25:49 dist-test-slave-pk3j krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:49.085301 2176 raft_consensus.cc:504] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b870f916dd5645ff9eca5bffe23c388d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } }
I20260620 14:25:49.085651 2176 leader_election.cc:302] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b870f916dd5645ff9eca5bffe23c388d; no voters:
I20260620 14:25:49.085963 2176 leader_election.cc:288] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:25:49.086264 2181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:49.086431 2181 raft_consensus.cc:686] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [term 1 LEADER]: Becoming Leader. State: Replica: b870f916dd5645ff9eca5bffe23c388d, State: Running, Role: LEADER
I20260620 14:25:49.086691 2181 consensus_queue.cc:237] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b870f916dd5645ff9eca5bffe23c388d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } }
I20260620 14:25:49.087132 2176 sys_catalog.cc:564] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:49.087761 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b870f916dd5645ff9eca5bffe23c388d. Latest consensus state: current_term: 1 leader_uuid: "b870f916dd5645ff9eca5bffe23c388d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b870f916dd5645ff9eca5bffe23c388d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } } }
I20260620 14:25:49.087899 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:49.089540 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b870f916dd5645ff9eca5bffe23c388d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b870f916dd5645ff9eca5bffe23c388d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43449 } } }
I20260620 14:25:49.089668 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:49.090585 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.076565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.0.98.62:43449)
0620 14:25:49.076978 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.076992 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.077025 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:25:49.077774 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.077799 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.078165 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.078482 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:49.079237 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.080686 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.081584 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.082293 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.082455 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.085070 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:49.085076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:49.085087 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:49.085110 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:49.087148 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.087616 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.087622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.087627 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.087764 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.088138 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.088142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.088143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.088616 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:49.088802 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.089695 (+ 893us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.089959 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":128,"threads_started":1}
W20260620 14:25:49.092233 2196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:25:49.092307 2196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:25:49.092386 2189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:49.093225 2189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:25:49.098263 2189 catalog_manager.cc:1347] Generated new cluster ID: 1637a33f9e1547d3b3fce762f0f0f284
I20260620 14:25:49.098317 2189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:25:49.105898 2189 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:49.106647 2189 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:49.122308 2189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d: Generated new TSK 0
I20260620 14:25:49.122848 2189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:25:49.151479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43449
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:49.255903 2202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:49.256158 2202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:49.256234 2202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:49.260095 2202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:49.260171 2202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:49.260238 2202 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:25:49.260280 2202 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:25:49.260344 2202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:25:49.265101 2202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:49.266026 2202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:49.266949 2202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:49.275489 2207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:49.275553 2210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:49.275516 2208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:49.275648 2202 server_base.cc:1034] running on GCE node
I20260620 14:25:49.276145 2202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:49.276702 2202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:49.277878 2202 hybrid_clock.cc:648] HybridClock initialized: now 1781965549277875 us; error 32 us; skew 500 ppm
Jun 20 14:25:49 dist-test-slave-pk3j krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:49.280339 2202 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:25:49.281111 2202 webserver.cc:460] Webserver started at http://127.0.98.1:44007/ using document root <none> and password file <none>
I20260620 14:25:49.281646 2202 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:49.281733 2202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:49.281932 2202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:49.283716 2202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "736280c3f5d142e09ebebbcdaec9ff50"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "fddb28f6dc0335502b3f1bd07d3fb6af"
server_key_iv: "14c0d44515d8854d029c99f10e1a07ba"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.284243 2202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "736280c3f5d142e09ebebbcdaec9ff50"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "fddb28f6dc0335502b3f1bd07d3fb6af"
server_key_iv: "14c0d44515d8854d029c99f10e1a07ba"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.287739 2202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:25:49.290124 2217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:49.291186 2202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:25:49.291308 2202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "736280c3f5d142e09ebebbcdaec9ff50"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "fddb28f6dc0335502b3f1bd07d3fb6af"
server_key_iv: "14c0d44515d8854d029c99f10e1a07ba"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.291405 2202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:49.307129 2202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:49.309654 2202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:49.309831 2202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:49.311089 2202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:49.311156 2202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:49.311215 2202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:49.311265 2202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:25:49 dist-test-slave-pk3j krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:49.322409 2202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35263
I20260620 14:25:49.322480 2332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35263 every 8 connection(s)
I20260620 14:25:49.323408 2202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:25:49.324626 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.311779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41877 (local address 127.0.98.62:43449)
0620 14:25:49.311931 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.311936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.312741 (+ 805us) server_negotiation.cc:407] Connection header received
0620 14:25:49.313941 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.313945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.313996 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.314087 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:49.315540 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.316463 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.317898 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.318136 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.318255 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.320810 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:49.320814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:49.320816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:49.320848 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:49.322460 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.323017 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.323020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.323021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.323092 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.323425 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.323428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.323429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.323621 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:49.323750 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.324352 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.324466 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:25:49.325350 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.312107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43449 (local address 127.0.98.1:41877)
0620 14:25:49.312563 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:49.312618 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:49.313721 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:49.314259 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:49.314270 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:49.314755 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:49.315365 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.315379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.316598 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.316602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.317763 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.317781 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.318297 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.318301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.318499 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.319073 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:49.319095 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:49.320653 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:49.322582 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.322603 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.322615 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.322906 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.323182 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.323185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.323187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.323333 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.323802 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:49.323806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:49.324087 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 14:25:49.324309 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 14:25:49.324581 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20260620 14:25:49.326511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 2202
I20260620 14:25:49.326558 2333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43449
I20260620 14:25:49.326661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:25:49.326678 2333 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:49.327035 392 external_mini_cluster.cc:1428] Setting key d7f102dcf6291f7a011531fa57159c85
I20260620 14:25:49.327217 2333 heartbeater.cc:511] Master 127.0.98.62:43449 requested a full tablet report, sending...
I20260620 14:25:49.328662 2130 ts_manager.cc:194] Registered new tserver with Master: 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263)
I20260620 14:25:49.329800 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41877
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:25:49.338761 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.331837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43449 (local address 127.0.98.1:32913)
0620 14:25:49.332048 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:49.332060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:49.332170 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:49.332401 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:49.332404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:49.332584 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:49.332664 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.332670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.333534 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.333537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.334115 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.334159 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.334817 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.334820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.334902 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.335378 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:49.335388 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:49.335686 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:49.337563 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.337565 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.337567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.337738 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.337984 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.337986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.337987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.338054 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.338377 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:49.338378 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:49.338441 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 14:25:49.338536 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:25:49.338628 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260620 14:25:49.338774 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.330538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32913 (local address 127.0.98.62:43449)
0620 14:25:49.330627 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.330630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.332129 (+ 1499us) server_negotiation.cc:407] Connection header received
0620 14:25:49.332241 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.332245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.332291 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.332370 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:49.332797 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.333417 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.334285 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.334691 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.334806 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.335823 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:49.335829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:49.335834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:49.335862 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:49.337464 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.337839 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.337842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.337843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.337908 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.338147 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.338152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.338157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.338310 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:49.338387 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.338591 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.338674 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:25:49.351606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43449
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:49.454849 2342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:49.455052 2342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:49.455096 2342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:49.458948 2342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:49.458994 2342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:49.459033 2342 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:25:49.459046 2342 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:25:49.459076 2342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:25:49.463860 2342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:49.464613 2342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:49.465497 2342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:49.473994 2347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:49.474002 2348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:49.474002 2350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:49.474334 2342 server_base.cc:1034] running on GCE node
I20260620 14:25:49.474699 2342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:49.475276 2342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:49.476468 2342 hybrid_clock.cc:648] HybridClock initialized: now 1781965549476460 us; error 29 us; skew 500 ppm
Jun 20 14:25:49 dist-test-slave-pk3j krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:49.479110 2342 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:25:49.479832 2342 webserver.cc:460] Webserver started at http://127.0.98.2:35605/ using document root <none> and password file <none>
I20260620 14:25:49.480355 2342 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:49.480441 2342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:49.480650 2342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:49.482666 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "30960957612b485aa63d69deec3fdbea"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "8f9881d8416c0c5b0d9daa1f605bc6fb"
server_key_iv: "0129657ccd61feeb91534385c97403d8"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.483206 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "30960957612b485aa63d69deec3fdbea"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "8f9881d8416c0c5b0d9daa1f605bc6fb"
server_key_iv: "0129657ccd61feeb91534385c97403d8"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.486662 2342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 14:25:49.489051 2357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:49.490170 2342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:25:49.490294 2342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "30960957612b485aa63d69deec3fdbea"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "8f9881d8416c0c5b0d9daa1f605bc6fb"
server_key_iv: "0129657ccd61feeb91534385c97403d8"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.490393 2342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:49.501495 2342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:49.503805 2342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:49.503978 2342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:49.505453 2342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:49.505511 2342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:49.505571 2342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:49.505604 2342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:25:49 dist-test-slave-pk3j krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:49.515959 2342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40961
I20260620 14:25:49.516011 2472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40961 every 8 connection(s)
I20260620 14:25:49.516877 2342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:25:49.518399 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.506281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46781 (local address 127.0.98.62:43449)
0620 14:25:49.506406 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.506410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.507282 (+ 872us) server_negotiation.cc:407] Connection header received
0620 14:25:49.508450 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.508456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.508520 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.508580 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:49.510308 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.510969 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.512005 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.512304 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.512418 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.514872 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:49.514878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:49.514882 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:49.514905 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:49.516517 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.517056 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.517059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.517060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.517126 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.517514 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.517519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.517523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.517745 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:49.517821 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.518205 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.518292 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:25:49.519026 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.506656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43449 (local address 127.0.98.2:46781)
0620 14:25:49.507110 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:49.507160 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:49.508193 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:49.509001 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:49.509014 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:49.509490 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:49.510148 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.510171 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.511099 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.511107 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.511879 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.511887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.512466 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.512470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.512618 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.513181 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:49.513241 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:49.514701 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:49.516616 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.516622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.516633 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.516972 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.517231 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.517235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.517237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.517397 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.517835 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:49.517839 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:49.518017 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 14:25:49.518146 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:25:49.518340 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260620 14:25:49.520030 2473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43449
I20260620 14:25:49.520190 2473 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:49.520622 2473 heartbeater.cc:511] Master 127.0.98.62:43449 requested a full tablet report, sending...
I20260620 14:25:49.521590 2129 ts_manager.cc:194] Registered new tserver with Master: 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961)
I20260620 14:25:49.522307 2129 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46781
I20260620 14:25:49.526681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 2342
I20260620 14:25:49.526844 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:25:49.527163 392 external_mini_cluster.cc:1428] Setting key a5b2abf26b46267127b780354a71ecd1
I20260620 14:25:49.528848 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.521939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43449 (local address 127.0.98.2:51727)
0620 14:25:49.522263 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:49.522276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:49.522369 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:49.522682 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:49.522685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:49.522873 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:49.522980 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.522985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.523953 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.523956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.524540 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.524548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.524920 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.524922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.524981 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.525381 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:49.525391 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:49.525674 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:49.527364 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.527366 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.527368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.527595 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.527863 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.527868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.527870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.527948 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.528442 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:49.528445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:49.528509 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:25:49.528591 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:25:49.528675 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":78,"threads_started":1}
I20260620 14:25:49.528944 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.522033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51727 (local address 127.0.98.62:43449)
0620 14:25:49.522134 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.522138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.522349 (+ 211us) server_negotiation.cc:407] Connection header received
0620 14:25:49.522509 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.522528 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.522579 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.522655 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:49.523095 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.523820 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.524630 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.524825 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.524899 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.525844 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:49.525849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:49.525855 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:49.525875 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:49.527270 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.527694 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.527698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.527699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.527780 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.528045 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.528048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.528049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.528329 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:49.528439 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.528665 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.528776 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:25:49.549527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43449
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:49.650427 2482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:49.650656 2482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:49.650731 2482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:49.654475 2482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:49.654546 2482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:49.654613 2482 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:25:49.654652 2482 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:25:49.654708 2482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:25:49.659468 2482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:49.660267 2482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:49.661095 2482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:49.669340 2487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:49.669348 2490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:49.669373 2488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:49.669544 2482 server_base.cc:1034] running on GCE node
I20260620 14:25:49.669935 2482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:49.670554 2482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:49.671720 2482 hybrid_clock.cc:648] HybridClock initialized: now 1781965549671699 us; error 33 us; skew 500 ppm
Jun 20 14:25:49 dist-test-slave-pk3j krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:49.674239 2482 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:25:49.674914 2482 webserver.cc:460] Webserver started at http://127.0.98.3:35641/ using document root <none> and password file <none>
I20260620 14:25:49.675348 2482 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:49.675419 2482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:49.675559 2482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:49.677328 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "20cf60db52364e62859e02f1156bde6d"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "835ea2624b90df911960f5704c88902e"
server_key_iv: "465784c9882cfed012779d194d845c77"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.677778 2482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "20cf60db52364e62859e02f1156bde6d"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "835ea2624b90df911960f5704c88902e"
server_key_iv: "465784c9882cfed012779d194d845c77"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.680821 2482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:25:49.682900 2497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:49.683977 2482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:25:49.684085 2482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "20cf60db52364e62859e02f1156bde6d"
format_stamp: "Formatted at 2026-06-20 14:25:49 on dist-test-slave-pk3j"
server_key: "835ea2624b90df911960f5704c88902e"
server_key_iv: "465784c9882cfed012779d194d845c77"
server_key_version: "encryptionkey@0"
I20260620 14:25:49.684170 2482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:49.696835 2482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:49.699658 2482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:49.699836 2482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:49.700992 2482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:49.701061 2482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:49.701126 2482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:49.701172 2482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:25:49 dist-test-slave-pk3j krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:49.711583 2482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38531
I20260620 14:25:49.711645 2612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38531 every 8 connection(s)
I20260620 14:25:49.712476 2482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:25:49.713575 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.701776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54897 (local address 127.0.98.62:43449)
0620 14:25:49.701893 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.701896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.702743 (+ 847us) server_negotiation.cc:407] Connection header received
0620 14:25:49.703537 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.703541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.703595 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.703693 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:49.705099 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.705965 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.707199 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.707430 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.707539 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.710157 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:49.710160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:49.710164 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:49.710193 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:49.711773 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.712192 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.712196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.712197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.712282 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.712556 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.712559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.712561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.712794 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:49.712898 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.713378 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.713469 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:25:49.713980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 2482
I20260620 14:25:49.714087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:25:49.714351 392 external_mini_cluster.cc:1428] Setting key a974884861baf5bb334adf5a66a2ba04
I20260620 14:25:49.714354 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.702209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43449 (local address 127.0.98.3:54897)
0620 14:25:49.702612 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:49.702646 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:49.703347 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:49.703915 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:49.703926 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:49.704381 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:49.704948 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.704962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.706118 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.706121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.707077 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.707087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.707581 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.707585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.707763 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.708270 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:49.708293 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:49.710005 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:49.711862 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.711866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.711875 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.712094 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.712364 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.712365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.712367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.712468 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.712899 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:49.712913 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:49.713140 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 14:25:49.713347 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 14:25:49.713562 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":121,"threads_started":1}
I20260620 14:25:49.715488 2613 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43449
I20260620 14:25:49.715607 2613 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:49.716053 2613 heartbeater.cc:511] Master 127.0.98.62:43449 requested a full tablet report, sending...
I20260620 14:25:49.717020 2128 ts_manager.cc:194] Registered new tserver with Master: 20cf60db52364e62859e02f1156bde6d (127.0.98.3:38531)
I20260620 14:25:49.717832 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54897
I20260620 14:25:49.723618 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.717493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43449 (local address 127.0.98.3:34485)
0620 14:25:49.717754 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:49.717768 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:49.717858 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:49.718096 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:49.718100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:49.718286 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:49.718402 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.718409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.719083 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.719086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.719617 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.719624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.720054 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.720057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.720140 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.720491 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:49.720501 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:49.720752 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:49.722273 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.722277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.722279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.722460 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.722721 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:49.722725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:49.722727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:49.722808 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:49.723142 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:49.723145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:49.723228 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 14:25:49.723335 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:25:49.723452 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260620 14:25:49.723619 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.717607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34485 (local address 127.0.98.62:43449)
0620 14:25:49.717710 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.717714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.717835 (+ 121us) server_negotiation.cc:407] Connection header received
0620 14:25:49.717944 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.717947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.717987 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.718078 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:49.718511 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.718971 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.719728 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.719950 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.720051 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.720876 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:49.720880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:49.720882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:49.720903 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:49.722167 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.722555 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.722560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.722562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.722637 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.722892 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.722897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.722899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.723052 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:49.723151 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.723395 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.723498 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:25:49.727807 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:25:49 dist-test-slave-pk3j krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:25:49 dist-test-slave-pk3j krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:49.750119 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.743173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.0.98.62:43449)
0620 14:25:49.743298 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.743301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.743312 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:25:49.743338 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.743340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.743368 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.743428 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:49.744011 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.744454 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.745172 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.745411 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.745484 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.747530 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:49.747533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:49.747535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:49.747553 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:49.748821 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.749139 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.749141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.749142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.749240 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:49.749467 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:49.749469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:49.749470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:49.749603 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:49.749691 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.749858 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.749930 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:25:49.752522 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:25:49.755342 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:25:49.767879 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.763054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40961 (local address 127.0.0.1:36816)
0620 14:25:49.763382 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:49.763397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:49.763486 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:49.764163 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:49.764168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:49.764190 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:49.764282 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.764288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.765179 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.765183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.766471 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.766481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.767349 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.767352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.767474 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.767513 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:25:49.767643 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:25:49.767752 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260620 14:25:49.767889 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.763244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.0.98.2:40961)
0620 14:25:49.763591 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.763602 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.763623 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:25:49.763692 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.763697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.763881 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.764028 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:49.764412 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.765064 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.766603 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.767231 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.767352 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.767521 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.767671 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.767758 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":111,"threads_started":1}
I20260620 14:25:49.768141 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.762438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38531 (local address 127.0.0.1:35092)
0620 14:25:49.762878 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:49.762920 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:49.763039 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:49.764212 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:49.764215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:49.764226 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:49.764295 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.764299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.765587 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.765591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.766752 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.766760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.767753 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.767770 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.767867 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.767890 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:25:49.767937 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:25:49.768006 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":27,"thread_start_us":66,"threads_started":1}
I20260620 14:25:49.768142 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.762616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35263 (local address 127.0.0.1:59838)
0620 14:25:49.763255 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:49.763267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:49.763342 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:49.764072 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:49.764077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:49.764098 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:49.764218 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.764224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.765613 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.765616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.766709 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:49.766717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.767839 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.767842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:49.767932 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.767951 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:25:49.767989 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 14:25:49.768034 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":47,"thread_start_us":71,"threads_started":1}
I20260620 14:25:49.768676 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.763025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.0.98.1:35263)
0620 14:25:49.763705 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.763718 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.763735 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:25:49.763792 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.763796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.763950 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.764069 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:49.764353 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.765469 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.766817 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.767620 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.768216 (+ 596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.768407 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.768456 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.768531 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":105,"threads_started":1}
I20260620 14:25:49.769381 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:49.762560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.0.98.3:38531)
0620 14:25:49.763822 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:49.763836 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:49.763858 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:25:49.763923 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:49.763927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:49.764103 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:49.764240 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:49.764359 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.765473 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.766878 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:49.767620 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:49.768956 (+ 1336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:49.769083 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:49.769146 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:25:49.769242 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":89,"threads_started":1}
I20260620 14:25:49.770591 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 6d931cd9ef7e4e5196553ca2c1c32d0d (DEFAULT_TABLE table=test-table [id=a03a84afbf894ed6a14c344d596cbd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:49.770939 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 6d931cd9ef7e4e5196553ca2c1c32d0d (DEFAULT_TABLE table=test-table [id=a03a84afbf894ed6a14c344d596cbd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:49.771368 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d931cd9ef7e4e5196553ca2c1c32d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:49.771615 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d931cd9ef7e4e5196553ca2c1c32d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:49.775475 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 6d931cd9ef7e4e5196553ca2c1c32d0d (DEFAULT_TABLE table=test-table [id=a03a84afbf894ed6a14c344d596cbd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:49.776397 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d931cd9ef7e4e5196553ca2c1c32d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:49.777485 2636 tablet_bootstrap.cc:492] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: Bootstrap starting.
I20260620 14:25:49.778275 2637 tablet_bootstrap.cc:492] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: Bootstrap starting.
I20260620 14:25:49.780385 2636 tablet_bootstrap.cc:654] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:49.780385 2637 tablet_bootstrap.cc:654] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:49.781221 2637 log.cc:824] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:49.781219 2636 log.cc:824] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:49.782438 2640 tablet_bootstrap.cc:492] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: Bootstrap starting.
I20260620 14:25:49.783001 2636 tablet_bootstrap.cc:492] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: No bootstrap required, opened a new log
I20260620 14:25:49.783001 2637 tablet_bootstrap.cc:492] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: No bootstrap required, opened a new log
I20260620 14:25:49.783175 2637 ts_tablet_manager.cc:1397] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:25:49.783175 2636 ts_tablet_manager.cc:1397] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 14:25:49.784355 2640 tablet_bootstrap.cc:654] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:49.785053 2640 log.cc:824] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:49.786597 2640 tablet_bootstrap.cc:492] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: No bootstrap required, opened a new log
I20260620 14:25:49.786774 2640 ts_tablet_manager.cc:1397] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:25:49.786863 2637 raft_consensus.cc:348] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } }
I20260620 14:25:49.786863 2636 raft_consensus.cc:348] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } }
I20260620 14:25:49.787117 2637 raft_consensus.cc:374] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:49.787117 2636 raft_consensus.cc:374] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:49.787171 2637 raft_consensus.cc:729] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30960957612b485aa63d69deec3fdbea, State: Initialized, Role: FOLLOWER
I20260620 14:25:49.787178 2636 raft_consensus.cc:729] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736280c3f5d142e09ebebbcdaec9ff50, State: Initialized, Role: FOLLOWER
I20260620 14:25:49.787611 2636 consensus_queue.cc:260] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } }
I20260620 14:25:49.787611 2637 consensus_queue.cc:260] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } }
I20260620 14:25:49.788517 2636 ts_tablet_manager.cc:1428] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:25:49.788555 2637 ts_tablet_manager.cc:1428] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260620 14:25:49.788535 2333 heartbeater.cc:503] Master 127.0.98.62:43449 was elected leader, sending a full tablet report...
I20260620 14:25:49.789312 2473 heartbeater.cc:503] Master 127.0.98.62:43449 was elected leader, sending a full tablet report...
I20260620 14:25:49.790462 2640 raft_consensus.cc:348] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } }
I20260620 14:25:49.790691 2640 raft_consensus.cc:374] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:49.790756 2640 raft_consensus.cc:729] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cf60db52364e62859e02f1156bde6d, State: Initialized, Role: FOLLOWER
I20260620 14:25:49.791157 2640 consensus_queue.cc:260] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } }
I20260620 14:25:49.791962 2613 heartbeater.cc:503] Master 127.0.98.62:43449 was elected leader, sending a full tablet report...
I20260620 14:25:49.792070 2640 ts_tablet_manager.cc:1428] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 14:25:49.824525 2334 tablet.cc:2369] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:25:49.963515 2614 tablet.cc:2369] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:25:50.017962 2474 tablet.cc:2369] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:25:50.092206 2644 raft_consensus.cc:482] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:25:50.092360 2644 raft_consensus.cc:504] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } }
I20260620 14:25:50.093487 2644 leader_election.cc:288] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263), 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961)
I20260620 14:25:50.097430 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.093691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35263 (local address 127.0.98.3:51875)
0620 14:25:50.093818 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:50.093831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:50.093943 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:50.094225 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:50.094228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:50.094259 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:50.094411 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.094421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.095334 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.095336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.096471 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.096480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.097119 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.097122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.097188 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.097224 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:25:50.097285 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:25:50.097344 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:25:50.097502 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.093815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51875 (local address 127.0.98.1:35263)
0620 14:25:50.093947 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.093951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.093966 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:25:50.094026 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.094030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.094074 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.094168 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:50.094518 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.095227 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.096591 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.096981 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.097104 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.097163 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.097328 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.097392 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:25:50.098167 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.094064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36749 (local address 127.0.98.2:40961)
0620 14:25:50.094195 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.094200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.094358 (+ 158us) server_negotiation.cc:407] Connection header received
0620 14:25:50.094507 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.094510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.094559 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.094633 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:50.095272 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.095917 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.097072 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.097678 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.097761 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.097801 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.098018 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.098084 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:25:50.098168 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.093955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40961 (local address 127.0.98.3:36749)
0620 14:25:50.094269 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:50.094302 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:50.094390 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:50.094649 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:50.094651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:50.094665 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:50.095164 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.095172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.096032 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.096036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.096956 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.096962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.097800 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.097805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.097899 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.097914 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:25:50.097973 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:25:50.098055 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260620 14:25:50.098268 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d931cd9ef7e4e5196553ca2c1c32d0d" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736280c3f5d142e09ebebbcdaec9ff50" is_pre_election: true
I20260620 14:25:50.098583 2285 raft_consensus.cc:2454] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 0.
I20260620 14:25:50.098701 2427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d931cd9ef7e4e5196553ca2c1c32d0d" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30960957612b485aa63d69deec3fdbea" is_pre_election: true
I20260620 14:25:50.098943 2427 raft_consensus.cc:2454] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 0.
I20260620 14:25:50.099052 2500 leader_election.cc:302] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cf60db52364e62859e02f1156bde6d, 736280c3f5d142e09ebebbcdaec9ff50; no voters:
I20260620 14:25:50.099313 2644 raft_consensus.cc:2781] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:25:50.099402 2644 raft_consensus.cc:482] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:25:50.099434 2644 raft_consensus.cc:3037] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:50.100390 2644 raft_consensus.cc:504] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } }
I20260620 14:25:50.100682 2644 leader_election.cc:288] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 election: Requested vote from peers 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263), 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961)
I20260620 14:25:50.101027 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d931cd9ef7e4e5196553ca2c1c32d0d" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736280c3f5d142e09ebebbcdaec9ff50"
I20260620 14:25:50.101042 2427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d931cd9ef7e4e5196553ca2c1c32d0d" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30960957612b485aa63d69deec3fdbea"
I20260620 14:25:50.101138 2427 raft_consensus.cc:3037] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:50.101138 2285 raft_consensus.cc:3037] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:50.102295 2285 raft_consensus.cc:2454] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 1.
I20260620 14:25:50.102295 2427 raft_consensus.cc:2454] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 1.
I20260620 14:25:50.102547 2500 leader_election.cc:302] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cf60db52364e62859e02f1156bde6d, 30960957612b485aa63d69deec3fdbea; no voters:
I20260620 14:25:50.102712 2644 raft_consensus.cc:2781] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:50.102906 2644 raft_consensus.cc:686] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 1 LEADER]: Becoming Leader. State: Replica: 20cf60db52364e62859e02f1156bde6d, State: Running, Role: LEADER
I20260620 14:25:50.103191 2644 consensus_queue.cc:237] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } }
I20260620 14:25:50.105976 2130 catalog_manager.cc:5495] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 20cf60db52364e62859e02f1156bde6d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "20cf60db52364e62859e02f1156bde6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:25:50 dist-test-slave-pk3j krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:50.146169 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.138584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43449 (local address 127.0.0.1:45062)
0620 14:25:50.138887 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:50.138899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:50.139015 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:50.139243 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:50.139245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:50.139615 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:50.139689 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.139695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.140435 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.140438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.141000 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.141008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.141527 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.141530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.141663 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.142147 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:50.142157 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:50.143458 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:50.144855 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:50.144862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:50.144866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:50.145067 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:50.145334 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:50.145336 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:50.145337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:50.145409 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:50.145748 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:50.145753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:50.145845 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 14:25:50.145932 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:25:50.146025 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":19,"thread_start_us":96,"threads_started":1}
I20260620 14:25:50.146186 2179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.138680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.0.98.62:43449)
0620 14:25:50.138826 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.138831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.138968 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:25:50.139102 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.139107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.139149 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.139228 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:50.139809 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.140303 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.141103 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.141425 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.141529 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.143579 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:50.143581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:50.143583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:50.143603 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:50.144767 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:50.145146 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:50.145148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:50.145150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:50.145252 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:50.145482 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:50.145484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:50.145485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:50.145634 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:50.145731 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.145990 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.146072 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":70}
I20260620 14:25:50.149681 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45062:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:25:50.150429 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:25:50.154902 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef5c3b44cf744499565978eea12e509 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a87bd4c1780347a4adcb2c4c963097e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:25:50.154899 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef5c3b44cf744499565978eea12e509 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a87bd4c1780347a4adcb2c4c963097e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:25:50.154991 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef5c3b44cf744499565978eea12e509 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a87bd4c1780347a4adcb2c4c963097e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:25:50.155604 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef5c3b44cf744499565978eea12e509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:50.155601 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef5c3b44cf744499565978eea12e509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:50.155697 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef5c3b44cf744499565978eea12e509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:50.158062 2640 tablet_bootstrap.cc:492] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: Bootstrap starting.
I20260620 14:25:50.158280 2636 tablet_bootstrap.cc:492] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: Bootstrap starting.
I20260620 14:25:50.158902 2640 tablet_bootstrap.cc:654] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:50.159009 2636 tablet_bootstrap.cc:654] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:50.159528 2637 tablet_bootstrap.cc:492] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: Bootstrap starting.
I20260620 14:25:50.160081 2640 tablet_bootstrap.cc:492] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: No bootstrap required, opened a new log
I20260620 14:25:50.160174 2640 ts_tablet_manager.cc:1397] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:25:50.160228 2637 tablet_bootstrap.cc:654] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:50.160555 2640 raft_consensus.cc:348] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.160660 2640 raft_consensus.cc:374] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:50.160707 2640 raft_consensus.cc:729] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cf60db52364e62859e02f1156bde6d, State: Initialized, Role: FOLLOWER
I20260620 14:25:50.160843 2640 consensus_queue.cc:260] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.161139 2644 tablet_replica.cc:439] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } }
I20260620 14:25:50.161312 2644 tablet_replica.cc:442] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:50.161379 2637 tablet_bootstrap.cc:492] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: No bootstrap required, opened a new log
I20260620 14:25:50.161140 2640 ts_tablet_manager.cc:1428] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:25:50.161448 2637 ts_tablet_manager.cc:1397] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:25:50.161880 2637 raft_consensus.cc:348] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.161998 2637 raft_consensus.cc:374] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:50.162068 2637 raft_consensus.cc:729] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30960957612b485aa63d69deec3fdbea, State: Initialized, Role: FOLLOWER
I20260620 14:25:50.162221 2636 tablet_bootstrap.cc:492] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: No bootstrap required, opened a new log
I20260620 14:25:50.162184 2637 consensus_queue.cc:260] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.162326 2636 ts_tablet_manager.cc:1397] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:25:50.162469 2637 ts_tablet_manager.cc:1428] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:25:50.162700 2643 tablet_replica.cc:439] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } }
I20260620 14:25:50.162757 2636 raft_consensus.cc:348] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.162842 2643 tablet_replica.cc:442] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:50.162868 2636 raft_consensus.cc:374] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:50.162909 2636 raft_consensus.cc:729] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736280c3f5d142e09ebebbcdaec9ff50, State: Initialized, Role: FOLLOWER
I20260620 14:25:50.163033 2636 consensus_queue.cc:260] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.163316 2636 ts_tablet_manager.cc:1428] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:50.163278 2642 tablet_replica.cc:439] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } }
I20260620 14:25:50.163400 2642 tablet_replica.cc:442] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:50.257975 2644 raft_consensus.cc:482] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:25:50.258080 2644 raft_consensus.cc:504] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.258490 2644 leader_election.cc:288] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263), 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961)
I20260620 14:25:50.258862 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef5c3b44cf744499565978eea12e509" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736280c3f5d142e09ebebbcdaec9ff50" is_pre_election: true
I20260620 14:25:50.258999 2285 raft_consensus.cc:2454] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 0.
I20260620 14:25:50.258993 2427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef5c3b44cf744499565978eea12e509" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30960957612b485aa63d69deec3fdbea" is_pre_election: true
I20260620 14:25:50.259166 2427 raft_consensus.cc:2454] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 0.
I20260620 14:25:50.259290 2500 leader_election.cc:302] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cf60db52364e62859e02f1156bde6d, 736280c3f5d142e09ebebbcdaec9ff50; no voters:
I20260620 14:25:50.259455 2644 raft_consensus.cc:2781] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:25:50.259514 2644 raft_consensus.cc:482] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:25:50.259560 2644 raft_consensus.cc:3037] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:50.260438 2644 raft_consensus.cc:504] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.260731 2644 leader_election.cc:288] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 election: Requested vote from peers 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263), 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961)
I20260620 14:25:50.261082 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef5c3b44cf744499565978eea12e509" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736280c3f5d142e09ebebbcdaec9ff50"
I20260620 14:25:50.261121 2427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef5c3b44cf744499565978eea12e509" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30960957612b485aa63d69deec3fdbea"
I20260620 14:25:50.261193 2285 raft_consensus.cc:3037] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:50.261260 2427 raft_consensus.cc:3037] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:50.262085 2285 raft_consensus.cc:2454] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 1.
I20260620 14:25:50.262130 2427 raft_consensus.cc:2454] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 1.
I20260620 14:25:50.262362 2500 leader_election.cc:302] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cf60db52364e62859e02f1156bde6d, 736280c3f5d142e09ebebbcdaec9ff50; no voters:
I20260620 14:25:50.262523 2644 raft_consensus.cc:2781] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:50.262606 2644 raft_consensus.cc:686] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [term 1 LEADER]: Becoming Leader. State: Replica: 20cf60db52364e62859e02f1156bde6d, State: Running, Role: LEADER
I20260620 14:25:50.262751 2644 consensus_queue.cc:237] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.263350 2646 tablet_replica.cc:439] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: TxnStatusTablet state changed. Reason: New leader 20cf60db52364e62859e02f1156bde6d. Latest consensus state: current_term: 1 leader_uuid: "20cf60db52364e62859e02f1156bde6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } }
I20260620 14:25:50.263463 2646 tablet_replica.cc:442] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:25:50.263686 2658 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:25:50.263917 2658 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:25:50.264247 2128 catalog_manager.cc:5495] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 20cf60db52364e62859e02f1156bde6d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "20cf60db52364e62859e02f1156bde6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } health_report { overall_health: HEALTHY } } }
I20260620 14:25:50.285771 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.281851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38531 (local address 127.0.0.1:35104)
0620 14:25:50.281989 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:50.282005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:50.282113 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:50.282466 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:50.282468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:50.282474 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:25:50.282556 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.282560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.283507 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.283509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.284041 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.284048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.284630 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.284632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.284710 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.284756 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:25:50.285522 (+ 766us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:25:50.285534 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:25:50.285585 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:25:50.285651 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:25:50.285931 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.281946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.0.98.3:38531)
0620 14:25:50.282132 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.282136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.282150 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:25:50.282296 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.282299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.282351 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.282454 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:50.282647 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.283409 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.284157 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.284474 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.284636 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.284920 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:50.285423 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:50.285527 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.285599 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.285650 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":47,"threads_started":1}
I20260620 14:25:50.290529 2427 raft_consensus.cc:1261] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Refusing update from remote peer 20cf60db52364e62859e02f1156bde6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:50.290614 2285 raft_consensus.cc:1261] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Refusing update from remote peer 20cf60db52364e62859e02f1156bde6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:50.291163 2644 consensus_queue.cc:1035] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:50.291316 2646 consensus_queue.cc:1035] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:50.295491 2642 tablet_replica.cc:439] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: TxnStatusTablet state changed. Reason: New leader 20cf60db52364e62859e02f1156bde6d. Latest consensus state: current_term: 1 leader_uuid: "20cf60db52364e62859e02f1156bde6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } }
I20260620 14:25:50.295598 2642 tablet_replica.cc:442] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:50.296201 2643 tablet_replica.cc:439] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: TxnStatusTablet state changed. Reason: New leader 20cf60db52364e62859e02f1156bde6d. Latest consensus state: current_term: 1 leader_uuid: "20cf60db52364e62859e02f1156bde6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } }
I20260620 14:25:50.296296 2643 tablet_replica.cc:442] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:50.297367 2646 tablet_replica.cc:439] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20cf60db52364e62859e02f1156bde6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } }
I20260620 14:25:50.297454 2646 tablet_replica.cc:442] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:25:50.297986 2643 tablet_replica.cc:439] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20cf60db52364e62859e02f1156bde6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } }
I20260620 14:25:50.298079 2643 tablet_replica.cc:442] T 2ef5c3b44cf744499565978eea12e509 P 30960957612b485aa63d69deec3fdbea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:50.298175 2662 mvcc.cc:205] Tried to move back new op lower bound from 7298930893984260096 to 7298930893877547008. Current Snapshot: MvccSnapshot[applied={T|T < 7298930893984260096}]
I20260620 14:25:50.298581 2664 mvcc.cc:205] Tried to move back new op lower bound from 7298930893984260096 to 7298930893877547008. Current Snapshot: MvccSnapshot[applied={T|T < 7298930893984260096}]
I20260620 14:25:50.302793 2646 tablet_replica.cc:439] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20cf60db52364e62859e02f1156bde6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } }
I20260620 14:25:50.302877 2646 tablet_replica.cc:442] T 2ef5c3b44cf744499565978eea12e509 P 20cf60db52364e62859e02f1156bde6d: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:25:50.304509 2642 tablet_replica.cc:439] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20cf60db52364e62859e02f1156bde6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } } }
I20260620 14:25:50.304606 2642 tablet_replica.cc:442] T 2ef5c3b44cf744499565978eea12e509 P 736280c3f5d142e09ebebbcdaec9ff50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:25:50.309435 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.305937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.0.98.3:38531)
0620 14:25:50.306057 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.306062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.306437 (+ 375us) server_negotiation.cc:407] Connection header received
0620 14:25:50.306534 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.306537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.306580 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.306657 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:50.306846 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.307558 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.308390 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.308705 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.308802 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.309001 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:50.309068 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:50.309136 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.309279 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.309322 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:25:50.318832 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.315769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42853 (local address 127.0.98.3:38531)
0620 14:25:50.315857 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.315862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.316024 (+ 162us) server_negotiation.cc:407] Connection header received
0620 14:25:50.316138 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.316151 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.316216 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.316301 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:50.316533 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.316974 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.317793 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.317988 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.318086 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.318382 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:50.318438 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:50.318509 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.318590 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.318676 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":24}
I20260620 14:25:50.318853 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.315688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38531 (local address 127.0.98.3:42853)
0620 14:25:50.315935 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:50.315955 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:50.316062 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:50.316306 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:50.316309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:50.316316 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:25:50.316427 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.316439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.317064 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.317069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.317682 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.317690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.318083 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.318086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.318217 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.318249 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:25:50.318522 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:25:50.318528 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:25:50.318583 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:25:50.318698 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":74,"thread_start_us":86,"threads_started":1}
I20260620 14:25:50.326050 2285 raft_consensus.cc:1261] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Refusing update from remote peer 20cf60db52364e62859e02f1156bde6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:50.326380 2646 consensus_queue.cc:1035] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:50.327675 2427 raft_consensus.cc:1261] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Refusing update from remote peer 20cf60db52364e62859e02f1156bde6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:50.328724 2646 consensus_queue.cc:1035] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:50.334637 2663 mvcc.cc:205] Tried to move back new op lower bound from 7298930894127517696 to 7298930893225185280. Current Snapshot: MvccSnapshot[applied={T|T < 7298930894127517696}]
I20260620 14:25:50.369831 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:50.370011 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:50.372179 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d: Sending DeleteTablet for 3 replicas of tablet 6d931cd9ef7e4e5196553ca2c1c32d0d
I20260620 14:25:50.372820 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d931cd9ef7e4e5196553ca2c1c32d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35092
I20260620 14:25:50.372987 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d931cd9ef7e4e5196553ca2c1c32d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59838
I20260620 14:25:50.373366 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d931cd9ef7e4e5196553ca2c1c32d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36816
I20260620 14:25:50.373452 2680 tablet_replica.cc:330] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: stopping tablet replica
I20260620 14:25:50.373687 2681 tablet_replica.cc:330] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: stopping tablet replica
I20260620 14:25:50.373742 2680 raft_consensus.cc:2229] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:25:50.373793 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:25:50.374055 2680 raft_consensus.cc:2258] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:25:50.374173 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:25:50.374310 2681 raft_consensus.cc:2229] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:50.374550 2681 raft_consensus.cc:2258] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:50.375228 2680 ts_tablet_manager.cc:1905] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:50.375738 2681 ts_tablet_manager.cc:1905] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:50.378157 2680 ts_tablet_manager.cc:1918] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:25:50.378249 2680 log.cc:1196] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/6d931cd9ef7e4e5196553ca2c1c32d0d
I20260620 14:25:50.378556 2680 ts_tablet_manager.cc:1939] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 20cf60db52364e62859e02f1156bde6d: Deleting consensus metadata
I20260620 14:25:50.378861 2681 ts_tablet_manager.cc:1918] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:25:50.379001 2681 log.cc:1196] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/6d931cd9ef7e4e5196553ca2c1c32d0d
I20260620 14:25:50.379343 2681 ts_tablet_manager.cc:1939] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 736280c3f5d142e09ebebbcdaec9ff50: Deleting consensus metadata
I20260620 14:25:50.379631 2114 catalog_manager.cc:4841] TS 20cf60db52364e62859e02f1156bde6d (127.0.98.3:38531): tablet 6d931cd9ef7e4e5196553ca2c1c32d0d (table test-table [id=a03a84afbf894ed6a14c344d596cbd49]) successfully deleted
I20260620 14:25:50.380107 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac97263608349a482f51a83f5d24cc3 (DEFAULT_TABLE table=test-table [id=32c089bda88944d7bfdd91b0d3ab4401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:50.380191 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac97263608349a482f51a83f5d24cc3 (DEFAULT_TABLE table=test-table [id=32c089bda88944d7bfdd91b0d3ab4401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:50.380334 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac97263608349a482f51a83f5d24cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:50.380411 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac97263608349a482f51a83f5d24cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:50.381544 2682 tablet_replica.cc:330] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: stopping tablet replica
I20260620 14:25:50.381857 2682 raft_consensus.cc:2229] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:50.382081 2682 raft_consensus.cc:2258] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:50.382220 2640 tablet_bootstrap.cc:492] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d: Bootstrap starting.
I20260620 14:25:50.382359 2636 tablet_bootstrap.cc:492] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50: Bootstrap starting.
I20260620 14:25:50.382962 2640 tablet_bootstrap.cc:654] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:50.383247 2636 tablet_bootstrap.cc:654] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:50.383970 2116 catalog_manager.cc:4841] TS 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263): tablet 6d931cd9ef7e4e5196553ca2c1c32d0d (table test-table [id=a03a84afbf894ed6a14c344d596cbd49]) successfully deleted
I20260620 14:25:50.384061 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac97263608349a482f51a83f5d24cc3 (DEFAULT_TABLE table=test-table [id=32c089bda88944d7bfdd91b0d3ab4401]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:50.384300 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac97263608349a482f51a83f5d24cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:50.384567 2682 ts_tablet_manager.cc:1905] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:50.384895 2636 tablet_bootstrap.cc:492] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50: No bootstrap required, opened a new log
I20260620 14:25:50.384984 2636 ts_tablet_manager.cc:1397] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:25:50.385489 2636 raft_consensus.cc:348] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.385597 2636 raft_consensus.cc:374] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:50.385636 2636 raft_consensus.cc:729] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736280c3f5d142e09ebebbcdaec9ff50, State: Initialized, Role: FOLLOWER
I20260620 14:25:50.385777 2636 consensus_queue.cc:260] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.386053 2636 ts_tablet_manager.cc:1428] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:50.386842 2637 tablet_bootstrap.cc:492] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea: Bootstrap starting.
I20260620 14:25:50.387811 2637 tablet_bootstrap.cc:654] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:50.388480 2682 ts_tablet_manager.cc:1918] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:25:50.388614 2682 log.cc:1196] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/6d931cd9ef7e4e5196553ca2c1c32d0d
I20260620 14:25:50.389236 2637 tablet_bootstrap.cc:492] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea: No bootstrap required, opened a new log
I20260620 14:25:50.389315 2637 ts_tablet_manager.cc:1397] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:25:50.389413 2682 ts_tablet_manager.cc:1939] T 6d931cd9ef7e4e5196553ca2c1c32d0d P 30960957612b485aa63d69deec3fdbea: Deleting consensus metadata
I20260620 14:25:50.389748 2637 raft_consensus.cc:348] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.389866 2637 raft_consensus.cc:374] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:50.389904 2637 raft_consensus.cc:729] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30960957612b485aa63d69deec3fdbea, State: Initialized, Role: FOLLOWER
I20260620 14:25:50.390050 2640 tablet_bootstrap.cc:492] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d: No bootstrap required, opened a new log
I20260620 14:25:50.390015 2637 consensus_queue.cc:260] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.390134 2640 ts_tablet_manager.cc:1397] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260620 14:25:50.390255 2637 ts_tablet_manager.cc:1428] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:50.390530 2640 raft_consensus.cc:348] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.390653 2640 raft_consensus.cc:374] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:50.390697 2640 raft_consensus.cc:729] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cf60db52364e62859e02f1156bde6d, State: Initialized, Role: FOLLOWER
I20260620 14:25:50.390820 2640 consensus_queue.cc:260] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.391023 2116 catalog_manager.cc:4841] TS 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961): tablet 6d931cd9ef7e4e5196553ca2c1c32d0d (table test-table [id=a03a84afbf894ed6a14c344d596cbd49]) successfully deleted
I20260620 14:25:50.391077 2640 ts_tablet_manager.cc:1428] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:50.523197 2643 raft_consensus.cc:482] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:25:50.523327 2643 raft_consensus.cc:504] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.524544 2643 leader_election.cc:288] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263), 20cf60db52364e62859e02f1156bde6d (127.0.98.3:38531)
I20260620 14:25:50.529575 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.525060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36875 (local address 127.0.98.1:35263)
0620 14:25:50.525199 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.525203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.525361 (+ 158us) server_negotiation.cc:407] Connection header received
0620 14:25:50.525545 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.525548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.525589 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.525632 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:50.526005 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.526786 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.528382 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.528954 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.529019 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.529075 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.529388 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.529457 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:25:50.529954 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.525320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46119 (local address 127.0.98.3:38531)
0620 14:25:50.525446 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.525450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.525716 (+ 266us) server_negotiation.cc:407] Connection header received
0620 14:25:50.525928 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.525932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.525982 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.526051 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:25:50.526479 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.527231 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.528691 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.529301 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.529381 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.529439 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.529766 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.529833 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:25:50.530203 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.524724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35263 (local address 127.0.98.2:36875)
0620 14:25:50.525286 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:50.525300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:50.525440 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:50.525783 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:50.525786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:50.525800 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:50.525905 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.525910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.526974 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.526977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.528250 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.528259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.529151 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.529154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.529268 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.529288 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:25:50.529907 (+ 619us) client_negotiation.cc:240] Negotiation successful
0620 14:25:50.529973 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260620 14:25:50.530368 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.524833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38531 (local address 127.0.98.2:46119)
0620 14:25:50.525612 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:50.525627 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:50.525785 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:50.526212 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:50.526215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:50.526232 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:25:50.526348 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.526356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.527390 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.527393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.528551 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.528562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.529530 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.529533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.529629 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.529649 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:25:50.530131 (+ 482us) client_negotiation.cc:240] Negotiation successful
0620 14:25:50.530202 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":34,"thread_start_us":94,"threads_started":1}
I20260620 14:25:50.530704 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac97263608349a482f51a83f5d24cc3" candidate_uuid: "30960957612b485aa63d69deec3fdbea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736280c3f5d142e09ebebbcdaec9ff50" is_pre_election: true
I20260620 14:25:50.530851 2285 raft_consensus.cc:2454] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30960957612b485aa63d69deec3fdbea in term 0.
I20260620 14:25:50.530853 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac97263608349a482f51a83f5d24cc3" candidate_uuid: "30960957612b485aa63d69deec3fdbea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20cf60db52364e62859e02f1156bde6d" is_pre_election: true
I20260620 14:25:50.531044 2567 raft_consensus.cc:2454] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30960957612b485aa63d69deec3fdbea in term 0.
I20260620 14:25:50.531205 2360 leader_election.cc:302] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30960957612b485aa63d69deec3fdbea, 736280c3f5d142e09ebebbcdaec9ff50; no voters:
I20260620 14:25:50.531539 2643 raft_consensus.cc:2781] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:25:50.531616 2643 raft_consensus.cc:482] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:25:50.531665 2643 raft_consensus.cc:3037] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:50.532853 2643 raft_consensus.cc:504] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.533313 2643 leader_election.cc:288] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [CANDIDATE]: Term 1 election: Requested vote from peers 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263), 20cf60db52364e62859e02f1156bde6d (127.0.98.3:38531)
I20260620 14:25:50.533527 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac97263608349a482f51a83f5d24cc3" candidate_uuid: "30960957612b485aa63d69deec3fdbea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736280c3f5d142e09ebebbcdaec9ff50"
I20260620 14:25:50.533638 2285 raft_consensus.cc:3037] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:50.533885 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac97263608349a482f51a83f5d24cc3" candidate_uuid: "30960957612b485aa63d69deec3fdbea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20cf60db52364e62859e02f1156bde6d"
I20260620 14:25:50.533982 2567 raft_consensus.cc:3037] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:50.534718 2285 raft_consensus.cc:2454] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30960957612b485aa63d69deec3fdbea in term 1.
I20260620 14:25:50.535069 2567 raft_consensus.cc:2454] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30960957612b485aa63d69deec3fdbea in term 1.
I20260620 14:25:50.535326 2358 leader_election.cc:302] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cf60db52364e62859e02f1156bde6d, 30960957612b485aa63d69deec3fdbea; no voters:
I20260620 14:25:50.535504 2643 raft_consensus.cc:2781] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:50.535712 2643 raft_consensus.cc:686] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 1 LEADER]: Becoming Leader. State: Replica: 30960957612b485aa63d69deec3fdbea, State: Running, Role: LEADER
I20260620 14:25:50.535931 2643 consensus_queue.cc:237] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:50.537905 2129 catalog_manager.cc:5495] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea reported cstate change: term changed from 0 to 1, leader changed from <none> to 30960957612b485aa63d69deec3fdbea (127.0.98.2). New cstate: current_term: 1 leader_uuid: "30960957612b485aa63d69deec3fdbea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20260620 14:25:50.583312 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.580032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.0.98.2:40961)
0620 14:25:50.580152 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.580158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.580171 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:25:50.580348 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.580350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.580397 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.580501 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:50.580718 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.581173 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.581963 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.582183 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.582290 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.582540 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:50.582964 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:50.583084 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.583140 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.583186 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:25:50.592119 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.589524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35107 (local address 127.0.98.3:38531)
0620 14:25:50.589658 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.589662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.589768 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:25:50.589844 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.589846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.589882 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.589941 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:50.590132 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.590575 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.591323 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.591513 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.591582 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.591823 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:50.591875 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:50.591925 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.591998 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.592032 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:25:50.592193 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.589416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38531 (local address 127.0.98.2:35107)
0620 14:25:50.589691 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:50.589703 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:50.589784 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:50.589961 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:50.589963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:50.589968 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:25:50.590050 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.590053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.590668 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.590670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.591232 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.591239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.591605 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.591607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.591686 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.591720 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:25:50.591944 (+ 224us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:25:50.591948 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:25:50.591993 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 14:25:50.592046 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":76,"threads_started":1}
I20260620 14:25:50.597662 2567 raft_consensus.cc:1261] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Refusing update from remote peer 30960957612b485aa63d69deec3fdbea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:50.597682 2285 raft_consensus.cc:1261] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Refusing update from remote peer 30960957612b485aa63d69deec3fdbea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:50.598140 2688 consensus_queue.cc:1035] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [LEADER]: Connected to new peer: Peer: permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:50.598332 2643 consensus_queue.cc:1035] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [LEADER]: Connected to new peer: Peer: permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:50.613744 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.610816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40961 (local address 127.0.98.3:60905)
0620 14:25:50.610974 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:50.610985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:50.611049 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:50.611232 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:50.611236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:50.611242 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:25:50.611319 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.611323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.611908 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.611912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.612520 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:50.612529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.613039 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.613043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:50.613122 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.613154 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:25:50.613477 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:25:50.613483 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:25:50.613567 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:25:50.613641 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 14:25:50.613740 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:50.610875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60905 (local address 127.0.98.2:40961)
0620 14:25:50.610984 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:50.610988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:50.611019 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:25:50.611103 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:50.611105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:50.611137 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:50.611218 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:50.611380 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.611799 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.612686 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:50.612930 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:50.613040 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:50.613299 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:50.613364 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:50.613455 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:50.613583 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:25:50.613627 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 14:25:50.614419 2664 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:25:52.167034 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:52.167173 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:52.169225 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d: Sending DeleteTablet for 3 replicas of tablet 6ac97263608349a482f51a83f5d24cc3
I20260620 14:25:52.169730 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac97263608349a482f51a83f5d24cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59838
I20260620 14:25:52.169768 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac97263608349a482f51a83f5d24cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35092
I20260620 14:25:52.169831 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ac97263608349a482f51a83f5d24cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36816
I20260620 14:25:52.170032 2719 tablet_replica.cc:330] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50: stopping tablet replica
I20260620 14:25:52.170421 2719 raft_consensus.cc:2229] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:52.170555 2719 raft_consensus.cc:2258] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:52.170936 2720 tablet_replica.cc:330] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d: stopping tablet replica
I20260620 14:25:52.171051 2720 raft_consensus.cc:2229] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:52.171146 2720 raft_consensus.cc:2258] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:52.171334 2719 ts_tablet_manager.cc:1905] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:52.171361 2721 tablet_replica.cc:330] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea: stopping tablet replica
I20260620 14:25:52.171473 2721 raft_consensus.cc:2229] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:25:52.171767 2721 raft_consensus.cc:2258] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:52.172705 2721 ts_tablet_manager.cc:1905] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:52.172838 2720 ts_tablet_manager.cc:1905] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:52.174104 2719 ts_tablet_manager.cc:1918] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:25:52.174177 2719 log.cc:1196] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/6ac97263608349a482f51a83f5d24cc3
I20260620 14:25:52.174409 2719 ts_tablet_manager.cc:1939] T 6ac97263608349a482f51a83f5d24cc3 P 736280c3f5d142e09ebebbcdaec9ff50: Deleting consensus metadata
I20260620 14:25:52.174993 2720 ts_tablet_manager.cc:1918] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:25:52.175071 2720 log.cc:1196] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/6ac97263608349a482f51a83f5d24cc3
I20260620 14:25:52.175149 2116 catalog_manager.cc:4841] TS 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263): tablet 6ac97263608349a482f51a83f5d24cc3 (table test-table [id=32c089bda88944d7bfdd91b0d3ab4401]) successfully deleted
I20260620 14:25:52.175199 2721 ts_tablet_manager.cc:1918] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:25:52.175253 2721 log.cc:1196] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/6ac97263608349a482f51a83f5d24cc3
I20260620 14:25:52.175347 2720 ts_tablet_manager.cc:1939] T 6ac97263608349a482f51a83f5d24cc3 P 20cf60db52364e62859e02f1156bde6d: Deleting consensus metadata
I20260620 14:25:52.175554 2721 ts_tablet_manager.cc:1939] T 6ac97263608349a482f51a83f5d24cc3 P 30960957612b485aa63d69deec3fdbea: Deleting consensus metadata
I20260620 14:25:52.176010 2114 catalog_manager.cc:4841] TS 20cf60db52364e62859e02f1156bde6d (127.0.98.3:38531): tablet 6ac97263608349a482f51a83f5d24cc3 (table test-table [id=32c089bda88944d7bfdd91b0d3ab4401]) successfully deleted
I20260620 14:25:52.176205 2116 catalog_manager.cc:4841] TS 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961): tablet 6ac97263608349a482f51a83f5d24cc3 (table test-table [id=32c089bda88944d7bfdd91b0d3ab4401]) successfully deleted
Jun 20 14:25:52 dist-test-slave-pk3j krb5kdc[2085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965552, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:25:52 dist-test-slave-pk3j krb5kdc[2085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965552, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:52.197837 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:52.190016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.0.98.62:43449)
0620 14:25:52.190216 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:52.190222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:52.190290 (+ 68us) server_negotiation.cc:407] Connection header received
0620 14:25:52.190402 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:52.190404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:52.190441 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:52.190507 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:52.191110 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:52.191639 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:52.192389 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:52.192608 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:52.192679 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:52.194957 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:52.194961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:52.194963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:52.194982 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:52.196391 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:52.196760 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:52.196762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:52.196764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:52.196832 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:52.197062 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:52.197063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:52.197065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:52.197287 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:52.197384 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:52.197527 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:25:52.197631 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260620 14:25:52.200014 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:25:52.200434 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:25:52.204383 2267 tablet_service.cc:1467] Processing CreateTablet for tablet 883d32f384d04aad836a2fb7d61cde8d (DEFAULT_TABLE table=test-table [id=295a3c34aba24f329e9c2653ac1630b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:52.204476 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 883d32f384d04aad836a2fb7d61cde8d (DEFAULT_TABLE table=test-table [id=295a3c34aba24f329e9c2653ac1630b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:52.204633 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883d32f384d04aad836a2fb7d61cde8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:52.204625 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 883d32f384d04aad836a2fb7d61cde8d (DEFAULT_TABLE table=test-table [id=295a3c34aba24f329e9c2653ac1630b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:52.204694 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883d32f384d04aad836a2fb7d61cde8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:52.204867 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 883d32f384d04aad836a2fb7d61cde8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:52.206667 2730 tablet_bootstrap.cc:492] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d: Bootstrap starting.
I20260620 14:25:52.207296 2731 tablet_bootstrap.cc:492] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea: Bootstrap starting.
I20260620 14:25:52.207433 2730 tablet_bootstrap.cc:654] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:52.208062 2731 tablet_bootstrap.cc:654] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:52.208360 2730 tablet_bootstrap.cc:492] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d: No bootstrap required, opened a new log
I20260620 14:25:52.208424 2730 ts_tablet_manager.cc:1397] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:25:52.208603 2732 tablet_bootstrap.cc:492] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50: Bootstrap starting.
I20260620 14:25:52.208866 2730 raft_consensus.cc:348] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.208947 2730 raft_consensus.cc:374] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:52.208967 2730 raft_consensus.cc:729] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cf60db52364e62859e02f1156bde6d, State: Initialized, Role: FOLLOWER
I20260620 14:25:52.209085 2730 consensus_queue.cc:260] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.209529 2732 tablet_bootstrap.cc:654] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:52.209683 2731 tablet_bootstrap.cc:492] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea: No bootstrap required, opened a new log
I20260620 14:25:52.209771 2731 ts_tablet_manager.cc:1397] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:25:52.209890 2730 ts_tablet_manager.cc:1428] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:52.210157 2731 raft_consensus.cc:348] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.210254 2731 raft_consensus.cc:374] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:52.210283 2731 raft_consensus.cc:729] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30960957612b485aa63d69deec3fdbea, State: Initialized, Role: FOLLOWER
I20260620 14:25:52.210418 2731 consensus_queue.cc:260] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.211032 2731 ts_tablet_manager.cc:1428] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:52.211236 2732 tablet_bootstrap.cc:492] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50: No bootstrap required, opened a new log
I20260620 14:25:52.211338 2732 ts_tablet_manager.cc:1397] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:25:52.211788 2732 raft_consensus.cc:348] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.211899 2732 raft_consensus.cc:374] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:52.211938 2732 raft_consensus.cc:729] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736280c3f5d142e09ebebbcdaec9ff50, State: Initialized, Role: FOLLOWER
I20260620 14:25:52.212065 2732 consensus_queue.cc:260] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.212401 2732 ts_tablet_manager.cc:1428] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:52.548293 2713 raft_consensus.cc:482] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:25:52.548417 2713 raft_consensus.cc:504] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.548735 2713 leader_election.cc:288] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263), 20cf60db52364e62859e02f1156bde6d (127.0.98.3:38531)
I20260620 14:25:52.549191 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883d32f384d04aad836a2fb7d61cde8d" candidate_uuid: "30960957612b485aa63d69deec3fdbea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736280c3f5d142e09ebebbcdaec9ff50" is_pre_election: true
I20260620 14:25:52.549178 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883d32f384d04aad836a2fb7d61cde8d" candidate_uuid: "30960957612b485aa63d69deec3fdbea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20cf60db52364e62859e02f1156bde6d" is_pre_election: true
I20260620 14:25:52.549352 2285 raft_consensus.cc:2454] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30960957612b485aa63d69deec3fdbea in term 0.
I20260620 14:25:52.549429 2567 raft_consensus.cc:2454] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30960957612b485aa63d69deec3fdbea in term 0.
I20260620 14:25:52.549718 2360 leader_election.cc:302] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30960957612b485aa63d69deec3fdbea, 736280c3f5d142e09ebebbcdaec9ff50; no voters:
I20260620 14:25:52.549875 2713 raft_consensus.cc:2781] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:25:52.549937 2713 raft_consensus.cc:482] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:25:52.549978 2713 raft_consensus.cc:3037] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:52.550872 2713 raft_consensus.cc:504] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.551170 2713 leader_election.cc:288] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [CANDIDATE]: Term 1 election: Requested vote from peers 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263), 20cf60db52364e62859e02f1156bde6d (127.0.98.3:38531)
I20260620 14:25:52.551438 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883d32f384d04aad836a2fb7d61cde8d" candidate_uuid: "30960957612b485aa63d69deec3fdbea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736280c3f5d142e09ebebbcdaec9ff50"
I20260620 14:25:52.551533 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "883d32f384d04aad836a2fb7d61cde8d" candidate_uuid: "30960957612b485aa63d69deec3fdbea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20cf60db52364e62859e02f1156bde6d"
I20260620 14:25:52.551578 2285 raft_consensus.cc:3037] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:52.551644 2567 raft_consensus.cc:3037] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:52.552670 2285 raft_consensus.cc:2454] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30960957612b485aa63d69deec3fdbea in term 1.
I20260620 14:25:52.552670 2567 raft_consensus.cc:2454] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30960957612b485aa63d69deec3fdbea in term 1.
I20260620 14:25:52.552942 2360 leader_election.cc:302] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30960957612b485aa63d69deec3fdbea, 736280c3f5d142e09ebebbcdaec9ff50; no voters:
I20260620 14:25:52.553088 2713 raft_consensus.cc:2781] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:52.553234 2713 raft_consensus.cc:686] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 1 LEADER]: Becoming Leader. State: Replica: 30960957612b485aa63d69deec3fdbea, State: Running, Role: LEADER
I20260620 14:25:52.553377 2713 consensus_queue.cc:237] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.554984 2127 catalog_manager.cc:5495] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea reported cstate change: term changed from 0 to 1, leader changed from <none> to 30960957612b485aa63d69deec3fdbea (127.0.98.2). New cstate: current_term: 1 leader_uuid: "30960957612b485aa63d69deec3fdbea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } health_report { overall_health: UNKNOWN } } }
I20260620 14:25:52.574265 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:52.571212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.0.98.2:40961)
0620 14:25:52.571454 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:52.571466 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:52.571480 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:25:52.571533 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:52.571536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:52.571571 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:52.571663 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:52.571895 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:52.572416 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:52.573286 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:52.573505 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:52.573613 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:52.573861 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:52.573929 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:52.574003 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:52.574083 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:25:52.574134 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260620 14:25:52.579283 2567 raft_consensus.cc:1261] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Refusing update from remote peer 30960957612b485aa63d69deec3fdbea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:52.579582 2285 raft_consensus.cc:1261] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Refusing update from remote peer 30960957612b485aa63d69deec3fdbea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:52.579685 2713 consensus_queue.cc:1035] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [LEADER]: Connected to new peer: Peer: permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:52.579905 2643 consensus_queue.cc:1035] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [LEADER]: Connected to new peer: Peer: permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:52.607838 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:52.607972 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:52.609684 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d: Sending DeleteTablet for 3 replicas of tablet 883d32f384d04aad836a2fb7d61cde8d
I20260620 14:25:52.610134 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 883d32f384d04aad836a2fb7d61cde8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59838
I20260620 14:25:52.610216 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 883d32f384d04aad836a2fb7d61cde8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36816
I20260620 14:25:52.610209 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet 883d32f384d04aad836a2fb7d61cde8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35092
I20260620 14:25:52.610296 2719 tablet_replica.cc:330] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50: stopping tablet replica
I20260620 14:25:52.610366 2720 tablet_replica.cc:330] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d: stopping tablet replica
I20260620 14:25:52.610394 2719 raft_consensus.cc:2229] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:52.610447 2720 raft_consensus.cc:2229] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:52.610481 2719 raft_consensus.cc:2258] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:52.610538 2720 raft_consensus.cc:2258] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:52.610651 2721 tablet_replica.cc:330] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea: stopping tablet replica
I20260620 14:25:52.610781 2721 raft_consensus.cc:2229] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:25:52.610939 2721 raft_consensus.cc:2258] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:52.610929 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:25:52.611277 2720 ts_tablet_manager.cc:1905] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:52.611449 2721 ts_tablet_manager.cc:1905] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:25:52.611467 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:25:52.613677 2721 ts_tablet_manager.cc:1918] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:25:52.613677 2720 ts_tablet_manager.cc:1918] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:25:52.613749 2720 log.cc:1196] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/883d32f384d04aad836a2fb7d61cde8d
I20260620 14:25:52.613749 2721 log.cc:1196] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/883d32f384d04aad836a2fb7d61cde8d
I20260620 14:25:52.613991 2720 ts_tablet_manager.cc:1939] T 883d32f384d04aad836a2fb7d61cde8d P 20cf60db52364e62859e02f1156bde6d: Deleting consensus metadata
I20260620 14:25:52.614012 2721 ts_tablet_manager.cc:1939] T 883d32f384d04aad836a2fb7d61cde8d P 30960957612b485aa63d69deec3fdbea: Deleting consensus metadata
I20260620 14:25:52.614550 2114 catalog_manager.cc:4841] TS 20cf60db52364e62859e02f1156bde6d (127.0.98.3:38531): tablet 883d32f384d04aad836a2fb7d61cde8d (table test-table [id=295a3c34aba24f329e9c2653ac1630b2]) successfully deleted
I20260620 14:25:52.614748 2116 catalog_manager.cc:4841] TS 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961): tablet 883d32f384d04aad836a2fb7d61cde8d (table test-table [id=295a3c34aba24f329e9c2653ac1630b2]) successfully deleted
I20260620 14:25:52.614883 2719 ts_tablet_manager.cc:1905] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:52.615970 2547 tablet_service.cc:1467] Processing CreateTablet for tablet acb78bac07d64f99b13522b805498a2a (DEFAULT_TABLE table=test-table [id=4a863bdb4fb94360b513fc04958b3ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:52.616001 2267 tablet_service.cc:1467] Processing CreateTablet for tablet acb78bac07d64f99b13522b805498a2a (DEFAULT_TABLE table=test-table [id=4a863bdb4fb94360b513fc04958b3ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:52.616197 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb78bac07d64f99b13522b805498a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:52.616263 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb78bac07d64f99b13522b805498a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:52.617429 2407 tablet_service.cc:1467] Processing CreateTablet for tablet acb78bac07d64f99b13522b805498a2a (DEFAULT_TABLE table=test-table [id=4a863bdb4fb94360b513fc04958b3ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:25:52.617594 2719 ts_tablet_manager.cc:1918] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:25:52.617642 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acb78bac07d64f99b13522b805498a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:52.617672 2719 log.cc:1196] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/883d32f384d04aad836a2fb7d61cde8d
I20260620 14:25:52.617903 2719 ts_tablet_manager.cc:1939] T 883d32f384d04aad836a2fb7d61cde8d P 736280c3f5d142e09ebebbcdaec9ff50: Deleting consensus metadata
I20260620 14:25:52.618139 2732 tablet_bootstrap.cc:492] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50: Bootstrap starting.
I20260620 14:25:52.618639 2730 tablet_bootstrap.cc:492] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d: Bootstrap starting.
I20260620 14:25:52.619058 2732 tablet_bootstrap.cc:654] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:52.619419 2730 tablet_bootstrap.cc:654] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:52.619831 2731 tablet_bootstrap.cc:492] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea: Bootstrap starting.
I20260620 14:25:52.620043 2116 catalog_manager.cc:4841] TS 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263): tablet 883d32f384d04aad836a2fb7d61cde8d (table test-table [id=295a3c34aba24f329e9c2653ac1630b2]) successfully deleted
I20260620 14:25:52.620270 2732 tablet_bootstrap.cc:492] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50: No bootstrap required, opened a new log
I20260620 14:25:52.620350 2732 ts_tablet_manager.cc:1397] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:25:52.620652 2731 tablet_bootstrap.cc:654] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:52.620780 2732 raft_consensus.cc:348] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.620900 2732 raft_consensus.cc:374] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:52.620947 2732 raft_consensus.cc:729] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736280c3f5d142e09ebebbcdaec9ff50, State: Initialized, Role: FOLLOWER
I20260620 14:25:52.621074 2732 consensus_queue.cc:260] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.621361 2732 ts_tablet_manager.cc:1428] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:52.621717 2731 tablet_bootstrap.cc:492] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea: No bootstrap required, opened a new log
I20260620 14:25:52.621770 2731 ts_tablet_manager.cc:1397] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:52.621961 2730 tablet_bootstrap.cc:492] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d: No bootstrap required, opened a new log
I20260620 14:25:52.622045 2730 ts_tablet_manager.cc:1397] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:25:52.622092 2731 raft_consensus.cc:348] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.622161 2731 raft_consensus.cc:374] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:52.622179 2731 raft_consensus.cc:729] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30960957612b485aa63d69deec3fdbea, State: Initialized, Role: FOLLOWER
I20260620 14:25:52.622287 2731 consensus_queue.cc:260] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.622401 2730 raft_consensus.cc:348] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.622504 2730 raft_consensus.cc:374] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:52.622515 2731 ts_tablet_manager.cc:1428] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:25:52.622534 2730 raft_consensus.cc:729] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20cf60db52364e62859e02f1156bde6d, State: Initialized, Role: FOLLOWER
I20260620 14:25:52.622634 2730 consensus_queue.cc:260] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.622856 2730 ts_tablet_manager.cc:1428] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:25:52.885053 2697 raft_consensus.cc:482] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:25:52.885187 2697 raft_consensus.cc:504] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.885619 2697 leader_election.cc:288] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263), 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961)
I20260620 14:25:52.886116 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb78bac07d64f99b13522b805498a2a" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736280c3f5d142e09ebebbcdaec9ff50" is_pre_election: true
I20260620 14:25:52.886148 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb78bac07d64f99b13522b805498a2a" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30960957612b485aa63d69deec3fdbea" is_pre_election: true
I20260620 14:25:52.886269 2287 raft_consensus.cc:2454] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 0.
I20260620 14:25:52.886303 2426 raft_consensus.cc:2454] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 0.
I20260620 14:25:52.886597 2500 leader_election.cc:302] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cf60db52364e62859e02f1156bde6d, 736280c3f5d142e09ebebbcdaec9ff50; no voters:
I20260620 14:25:52.886790 2697 raft_consensus.cc:2781] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:25:52.886866 2697 raft_consensus.cc:482] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:25:52.886893 2697 raft_consensus.cc:3037] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:52.887992 2697 raft_consensus.cc:504] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.888315 2697 leader_election.cc:288] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 election: Requested vote from peers 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263), 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961)
I20260620 14:25:52.888604 2287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb78bac07d64f99b13522b805498a2a" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "736280c3f5d142e09ebebbcdaec9ff50"
I20260620 14:25:52.888718 2287 raft_consensus.cc:3037] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:52.888713 2426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acb78bac07d64f99b13522b805498a2a" candidate_uuid: "20cf60db52364e62859e02f1156bde6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30960957612b485aa63d69deec3fdbea"
I20260620 14:25:52.888834 2426 raft_consensus.cc:3037] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:52.889868 2287 raft_consensus.cc:2454] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 1.
I20260620 14:25:52.890026 2426 raft_consensus.cc:2454] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20cf60db52364e62859e02f1156bde6d in term 1.
I20260620 14:25:52.890209 2500 leader_election.cc:302] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20cf60db52364e62859e02f1156bde6d, 736280c3f5d142e09ebebbcdaec9ff50; no voters:
I20260620 14:25:52.890391 2697 raft_consensus.cc:2781] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:52.890511 2697 raft_consensus.cc:686] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 1 LEADER]: Becoming Leader. State: Replica: 20cf60db52364e62859e02f1156bde6d, State: Running, Role: LEADER
I20260620 14:25:52.890641 2697 consensus_queue.cc:237] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } }
I20260620 14:25:52.892040 2129 catalog_manager.cc:5495] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 20cf60db52364e62859e02f1156bde6d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "20cf60db52364e62859e02f1156bde6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20cf60db52364e62859e02f1156bde6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38531 } health_report { overall_health: HEALTHY } } }
I20260620 14:25:52.913966 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:52.911181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.0.98.3:38531)
0620 14:25:52.911482 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:52.911489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:52.911501 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:25:52.911535 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:52.911537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:52.911574 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:52.911652 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:25:52.911839 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:52.912333 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:52.913092 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:52.913354 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:52.913429 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:52.913620 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:25:52.913685 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:25:52.913736 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:52.913826 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:25:52.913866 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":138,"threads_started":1}
I20260620 14:25:52.919126 2287 raft_consensus.cc:1261] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Refusing update from remote peer 20cf60db52364e62859e02f1156bde6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:52.919165 2426 raft_consensus.cc:1261] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Refusing update from remote peer 20cf60db52364e62859e02f1156bde6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:25:52.919492 2700 consensus_queue.cc:1035] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "30960957612b485aa63d69deec3fdbea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:25:52.919642 2697 consensus_queue.cc:1035] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "736280c3f5d142e09ebebbcdaec9ff50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:25:52.930794 2698 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:25:54.475471 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:54.475634 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:25:54.477859 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b870f916dd5645ff9eca5bffe23c388d: Sending DeleteTablet for 3 replicas of tablet acb78bac07d64f99b13522b805498a2a
I20260620 14:25:54.478334 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet acb78bac07d64f99b13522b805498a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59838
I20260620 14:25:54.478371 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet acb78bac07d64f99b13522b805498a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36816
I20260620 14:25:54.478585 2547 tablet_service.cc:1514] Processing DeleteTablet for tablet acb78bac07d64f99b13522b805498a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:25:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35092
I20260620 14:25:54.478655 2772 tablet_replica.cc:330] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50: stopping tablet replica
I20260620 14:25:54.478787 2772 raft_consensus.cc:2229] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:54.478907 2771 tablet_replica.cc:330] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea: stopping tablet replica
I20260620 14:25:54.478911 2773 tablet_replica.cc:330] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d: stopping tablet replica
I20260620 14:25:54.478935 2772 raft_consensus.cc:2258] T acb78bac07d64f99b13522b805498a2a P 736280c3f5d142e09ebebbcdaec9ff50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:54.479100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 2202
I20260620 14:25:54.480268 2771 raft_consensus.cc:2229] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:25:54.480360 2771 raft_consensus.cc:2258] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:54.480926 2773 raft_consensus.cc:2229] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:25:54.481031 2771 ts_tablet_manager.cc:1905] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:54.481124 2773 raft_consensus.cc:2258] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:25:54.481676 2773 ts_tablet_manager.cc:1905] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:25:54.483608 2773 ts_tablet_manager.cc:1918] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:25:54.483691 2773 log.cc:1196] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/acb78bac07d64f99b13522b805498a2a
I20260620 14:25:54.483925 2773 ts_tablet_manager.cc:1939] T acb78bac07d64f99b13522b805498a2a P 20cf60db52364e62859e02f1156bde6d: Deleting consensus metadata
I20260620 14:25:54.484438 2771 ts_tablet_manager.cc:1918] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:25:54.484519 2771 log.cc:1196] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/acb78bac07d64f99b13522b805498a2a
I20260620 14:25:54.484488 2114 catalog_manager.cc:4841] TS 20cf60db52364e62859e02f1156bde6d (127.0.98.3:38531): tablet acb78bac07d64f99b13522b805498a2a (table test-table [id=4a863bdb4fb94360b513fc04958b3ae5]) successfully deleted
I20260620 14:25:54.484853 2771 ts_tablet_manager.cc:1939] T acb78bac07d64f99b13522b805498a2a P 30960957612b485aa63d69deec3fdbea: Deleting consensus metadata
I20260620 14:25:54.485569 2116 catalog_manager.cc:4841] TS 30960957612b485aa63d69deec3fdbea (127.0.98.2:40961): tablet acb78bac07d64f99b13522b805498a2a (table test-table [id=4a863bdb4fb94360b513fc04958b3ae5]) successfully deleted
W20260620 14:25:54.486379 2116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35263 (error 108)
I20260620 14:25:54.486491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 2342
I20260620 14:25:54.488168 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:54.487704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35263 (local address 127.0.0.1:32994)
0620 14:25:54.487959 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:54.488042 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
W20260620 14:25:54.488416 2116 catalog_manager.cc:4584] TS 736280c3f5d142e09ebebbcdaec9ff50 (127.0.98.1:35263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet acb78bac07d64f99b13522b805498a2a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35263: connect: Connection refused (error 111)
I20260620 14:25:54.494073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 2482
I20260620 14:25:54.500368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 2098
2026-06-20T14:25:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6737 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:25:54 dist-test-slave-pk3j krb5kdc[2778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:25:54 dist-test-slave-pk3j krb5kdc[2778](info): set up 2 sockets
Jun 20 14:25:54 dist-test-slave-pk3j krb5kdc[2778](info): commencing operation
krb5kdc: starting...
W20260620 14:25:55.509017 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:25:55 dist-test-slave-pk3j krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:25:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:25:55.615319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:55.714891 2791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:55.715085 2791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:55.715137 2791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:55.718894 2791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:55.718932 2791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:55.718950 2791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:55.718964 2791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:55.718979 2791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:55.723749 2791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:55.724504 2791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:55.725296 2791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:55.733027 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:55.733035 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:55.733101 2791 server_base.cc:1034] running on GCE node
W20260620 14:25:55.733035 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:55.733683 2791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:55.734583 2791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:55.735759 2791 hybrid_clock.cc:648] HybridClock initialized: now 1781965555735743 us; error 33 us; skew 500 ppm
Jun 20 14:25:55 dist-test-slave-pk3j krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:55.738225 2791 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:25:55.739032 2791 webserver.cc:460] Webserver started at http://127.0.98.62:35445/ using document root <none> and password file <none>
I20260620 14:25:55.739562 2791 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:55.739652 2791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:55.739887 2791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:55.741707 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "b2c54ee4c4594993bc35fd607c956650"
format_stamp: "Formatted at 2026-06-20 14:25:55 on dist-test-slave-pk3j"
server_key: "7ed155fa1a9405ebd4c86701c8301944"
server_key_iv: "16ad1cae8ace66d0598f6b4f798e16fe"
server_key_version: "encryptionkey@0"
I20260620 14:25:55.742247 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "b2c54ee4c4594993bc35fd607c956650"
format_stamp: "Formatted at 2026-06-20 14:25:55 on dist-test-slave-pk3j"
server_key: "7ed155fa1a9405ebd4c86701c8301944"
server_key_iv: "16ad1cae8ace66d0598f6b4f798e16fe"
server_key_version: "encryptionkey@0"
I20260620 14:25:55.745507 2791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:25:55.747684 2806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:55.748679 2791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:55.748809 2791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "b2c54ee4c4594993bc35fd607c956650"
format_stamp: "Formatted at 2026-06-20 14:25:55 on dist-test-slave-pk3j"
server_key: "7ed155fa1a9405ebd4c86701c8301944"
server_key_iv: "16ad1cae8ace66d0598f6b4f798e16fe"
server_key_version: "encryptionkey@0"
I20260620 14:25:55.748907 2791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:55.762681 2791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:55.765059 2791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:55.765311 2791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:55.771734 2791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34079
I20260620 14:25:55.771760 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34079 every 8 connection(s)
I20260620 14:25:55.772529 2791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:55.774931 2859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:55.779738 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650: Bootstrap starting.
I20260620 14:25:55.780412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 2791
I20260620 14:25:55.780491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:55.780735 392 external_mini_cluster.cc:1428] Setting key 54fb7fd030be2fc1fee24d2be21a336e
I20260620 14:25:55.782229 2859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:55.782897 2859 log.cc:824] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:55.784653 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650: No bootstrap required, opened a new log
Jun 20 14:25:55 dist-test-slave-pk3j krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:55.788859 2859 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c54ee4c4594993bc35fd607c956650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } }
I20260620 14:25:55.789134 2859 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:55.789228 2859 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c54ee4c4594993bc35fd607c956650, State: Initialized, Role: FOLLOWER
I20260620 14:25:55.789690 2859 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c54ee4c4594993bc35fd607c956650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } }
I20260620 14:25:55.789845 2859 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:25:55.789909 2859 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:25:55.790005 2859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:55.791023 2859 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c54ee4c4594993bc35fd607c956650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } }
I20260620 14:25:55.791368 2859 leader_election.cc:302] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2c54ee4c4594993bc35fd607c956650; no voters:
I20260620 14:25:55.791718 2859 leader_election.cc:288] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:25:55.791754 2864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:55.792039 2864 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [term 1 LEADER]: Becoming Leader. State: Replica: b2c54ee4c4594993bc35fd607c956650, State: Running, Role: LEADER
I20260620 14:25:55.792320 2864 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c54ee4c4594993bc35fd607c956650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } }
I20260620 14:25:55.792932 2859 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:55.793407 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2c54ee4c4594993bc35fd607c956650. Latest consensus state: current_term: 1 leader_uuid: "b2c54ee4c4594993bc35fd607c956650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c54ee4c4594993bc35fd607c956650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } } }
I20260620 14:25:55.793581 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:55.793841 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2c54ee4c4594993bc35fd607c956650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c54ee4c4594993bc35fd607c956650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34079 } } }
I20260620 14:25:55.793987 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:55.794221 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:55.781756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.0.98.62:34079)
0620 14:25:55.782114 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:55.782127 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:55.782157 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:25:55.782743 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:55.782759 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:55.783078 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:55.783385 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:55.784121 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:55.785030 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:55.786218 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:55.786657 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:55.786767 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:55.789466 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:55.789472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:55.789485 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:55.789513 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:55.791393 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:55.791833 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:55.791838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:55.791841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:55.791937 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:55.792218 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:55.792221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:55.792223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:55.792620 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:55.792817 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:55.793067 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 14:25:55.793369 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
W20260620 14:25:55.797859 2880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:25:55.797936 2880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:25:55.798018 2870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:55.798779 2870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:25:55.803685 2870 catalog_manager.cc:1347] Generated new cluster ID: 6316549e668840fbbbc56b3e032215eb
I20260620 14:25:55.803745 2870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:25:55.822856 2870 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:55.823920 2870 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:55.835042 2870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2c54ee4c4594993bc35fd607c956650: Generated new TSK 0
I20260620 14:25:55.835700 2870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:25:55.869931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34079
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:55.974640 2885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:55.974856 2885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:55.974895 2885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:55.978650 2885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:55.978690 2885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:55.978740 2885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:25:55.984078 2885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:55.984956 2885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:55.985905 2885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:55.994415 2891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:55.994464 2893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:55.994437 2890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:55.994709 2885 server_base.cc:1034] running on GCE node
I20260620 14:25:55.995103 2885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:55.995592 2885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:55.996740 2885 hybrid_clock.cc:648] HybridClock initialized: now 1781965555996723 us; error 28 us; skew 500 ppm
Jun 20 14:25:55 dist-test-slave-pk3j krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:55.999307 2885 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:25:56.000077 2885 webserver.cc:460] Webserver started at http://127.0.98.1:35437/ using document root <none> and password file <none>
I20260620 14:25:56.000541 2885 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:56.000593 2885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:56.000734 2885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:56.002564 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "d64c6d1c7c5244aba87e5c2213b3605a"
format_stamp: "Formatted at 2026-06-20 14:25:55 on dist-test-slave-pk3j"
server_key: "0fde43ca12d3cad3d418926d5388b807"
server_key_iv: "9e3ac178750a0d97c25e6caf61272a89"
server_key_version: "encryptionkey@0"
I20260620 14:25:56.003039 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d64c6d1c7c5244aba87e5c2213b3605a"
format_stamp: "Formatted at 2026-06-20 14:25:55 on dist-test-slave-pk3j"
server_key: "0fde43ca12d3cad3d418926d5388b807"
server_key_iv: "9e3ac178750a0d97c25e6caf61272a89"
server_key_version: "encryptionkey@0"
I20260620 14:25:56.006369 2885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:25:56.008548 2900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:56.009716 2885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:25:56.009837 2885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "d64c6d1c7c5244aba87e5c2213b3605a"
format_stamp: "Formatted at 2026-06-20 14:25:55 on dist-test-slave-pk3j"
server_key: "0fde43ca12d3cad3d418926d5388b807"
server_key_iv: "9e3ac178750a0d97c25e6caf61272a89"
server_key_version: "encryptionkey@0"
I20260620 14:25:56.009948 2885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:56.020860 2885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:56.023458 2885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:56.023636 2885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:56.024137 2885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:56.024919 2885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:56.024982 2885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:56.025048 2885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:56.025089 2885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:56.033895 2885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36497
I20260620 14:25:56.033913 3013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36497 every 8 connection(s)
I20260620 14:25:56.034711 2885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:25:56.034987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 2885
I20260620 14:25:56.035115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:25:56.035442 392 external_mini_cluster.cc:1428] Setting key 25f469e038f9e0f9fe32b84779a2922d
Jun 20 14:25:56 dist-test-slave-pk3j krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:25:56.047638 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:56.036492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42603 (local address 127.0.98.62:34079)
0620 14:25:56.036619 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:56.036624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:56.037551 (+ 927us) server_negotiation.cc:407] Connection header received
0620 14:25:56.038860 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:56.038863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:56.038911 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:56.038996 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:56.040049 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.040553 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.041677 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.041896 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.042010 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:56.044252 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:56.044258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:56.044264 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:56.044286 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:56.045664 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:56.046206 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:56.046212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:56.046215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:56.046296 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:56.046625 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:56.046629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:56.046631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:56.046801 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:56.046910 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:56.047433 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 14:25:56.047525 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:25:56.048624 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:56.036959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34079 (local address 127.0.98.1:42603)
0620 14:25:56.037426 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:56.037464 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:56.038648 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:56.039166 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:56.039173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:56.039480 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:56.039891 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:56.039904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.040688 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.040691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:56.041563 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:56.041572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.042046 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.042050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:56.042197 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:56.042734 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:56.042749 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:56.044108 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:56.045780 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:56.045784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:56.045794 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:56.046093 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:56.046389 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:56.046393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:56.046396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:56.046529 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:56.046908 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:56.046915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:56.047190 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 14:25:56.047397 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 14:25:56.047719 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":159,"threads_started":1}
I20260620 14:25:56.049762 3014 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34079
I20260620 14:25:56.049966 3014 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:56.050392 3014 heartbeater.cc:511] Master 127.0.98.62:34079 requested a full tablet report, sending...
I20260620 14:25:56.051873 2823 ts_manager.cc:194] Registered new tserver with Master: d64c6d1c7c5244aba87e5c2213b3605a (127.0.98.1:36497)
I20260620 14:25:56.053066 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:25:56.059561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34079
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:56.159384 3019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:56.159585 3019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:56.159622 3019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:56.163319 3019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:56.163363 3019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:56.163414 3019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:25:56.168134 3019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:56.168931 3019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:56.169770 3019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:56.179143 3025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:56.179152 3024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:56.179160 3027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:56.179270 3019 server_base.cc:1034] running on GCE node
I20260620 14:25:56.179817 3019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:56.180321 3019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:56.181496 3019 hybrid_clock.cc:648] HybridClock initialized: now 1781965556181484 us; error 29 us; skew 500 ppm
Jun 20 14:25:56 dist-test-slave-pk3j krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:56.183862 3019 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:25:56.184619 3019 webserver.cc:460] Webserver started at http://127.0.98.2:39159/ using document root <none> and password file <none>
I20260620 14:25:56.185093 3019 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:56.185192 3019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:56.185398 3019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:56.187193 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "b98b3aeebfdb4d7588edce96f930341f"
format_stamp: "Formatted at 2026-06-20 14:25:56 on dist-test-slave-pk3j"
server_key: "8b42bcef03b3f11eafd1b4922cdbd4e2"
server_key_iv: "93f543217af58b1e7f46abaa04fbbb19"
server_key_version: "encryptionkey@0"
I20260620 14:25:56.187686 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b98b3aeebfdb4d7588edce96f930341f"
format_stamp: "Formatted at 2026-06-20 14:25:56 on dist-test-slave-pk3j"
server_key: "8b42bcef03b3f11eafd1b4922cdbd4e2"
server_key_iv: "93f543217af58b1e7f46abaa04fbbb19"
server_key_version: "encryptionkey@0"
I20260620 14:25:56.190923 3019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 14:25:56.193176 3034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:56.194459 3019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:25:56.194589 3019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "b98b3aeebfdb4d7588edce96f930341f"
format_stamp: "Formatted at 2026-06-20 14:25:56 on dist-test-slave-pk3j"
server_key: "8b42bcef03b3f11eafd1b4922cdbd4e2"
server_key_iv: "93f543217af58b1e7f46abaa04fbbb19"
server_key_version: "encryptionkey@0"
I20260620 14:25:56.194684 3019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:56.212728 3019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:56.215509 3019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:56.215682 3019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:56.216205 3019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:56.216929 3019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:56.216997 3019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:56.217059 3019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:56.217108 3019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:56.225160 3019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41035
I20260620 14:25:56.225188 3147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41035 every 8 connection(s)
I20260620 14:25:56.225978 3019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:25:56 dist-test-slave-pk3j krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:56.234644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 3019
I20260620 14:25:56.234767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:25:56.235008 392 external_mini_cluster.cc:1428] Setting key a16896c52999db3485fb9eb806f1fec8
I20260620 14:25:56.238765 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:56.227963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51399 (local address 127.0.98.62:34079)
0620 14:25:56.228086 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:56.228090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:56.228656 (+ 566us) server_negotiation.cc:407] Connection header received
0620 14:25:56.229861 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:56.229863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:56.229907 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:56.229977 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:56.230925 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.231424 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.232413 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.232788 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.232871 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:56.235115 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:56.235118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:56.235121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:56.235148 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:56.236770 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:56.237269 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:56.237275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:56.237280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:56.237364 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:56.237673 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:56.237679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:56.237684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:56.237915 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:56.238016 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:56.238574 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 14:25:56.238658 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:25:56.239738 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:56.228131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34079 (local address 127.0.98.2:51399)
0620 14:25:56.228519 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:56.228555 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:56.229683 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:56.230122 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:56.230128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:56.230424 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:56.230816 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:56.230827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.231522 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.231524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:56.232332 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:56.232339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.232903 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.232906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:56.233040 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:56.233591 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:56.233605 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:56.234960 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:56.236889 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:56.236894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:56.236904 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:56.237149 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:56.237458 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:56.237460 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:56.237462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:56.237578 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:56.238037 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:56.238046 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:56.238331 (+ 285us) client_negotiation.cc:769] Sending connection context
0620 14:25:56.238519 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 14:25:56.238821 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
I20260620 14:25:56.240856 3148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34079
I20260620 14:25:56.241055 3148 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:56.241513 3148 heartbeater.cc:511] Master 127.0.98.62:34079 requested a full tablet report, sending...
I20260620 14:25:56.242468 2823 ts_manager.cc:194] Registered new tserver with Master: b98b3aeebfdb4d7588edce96f930341f (127.0.98.2:41035)
I20260620 14:25:56.243003 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51399
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:25:56.260166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34079
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:56.360260 3153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:56.360517 3153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:56.360585 3153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:56.364506 3153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:56.364568 3153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:56.364661 3153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:25:56.369483 3153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:56.370287 3153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:56.371124 3153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:56.379357 3159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:56.379331 3158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:56.379395 3153 server_base.cc:1034] running on GCE node
W20260620 14:25:56.379360 3161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:56.379813 3153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:56.380339 3153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:56.381484 3153 hybrid_clock.cc:648] HybridClock initialized: now 1781965556381458 us; error 35 us; skew 500 ppm
Jun 20 14:25:56 dist-test-slave-pk3j krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:56.383797 3153 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:25:56.384598 3153 webserver.cc:460] Webserver started at http://127.0.98.3:37615/ using document root <none> and password file <none>
I20260620 14:25:56.385056 3153 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:56.385105 3153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:56.385342 3153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:56.387149 3153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "2611f7f43ff144129e3dcdd310ce89b0"
format_stamp: "Formatted at 2026-06-20 14:25:56 on dist-test-slave-pk3j"
server_key: "edbe90e830360a83034dc74baf586bd4"
server_key_iv: "6bf0c791c01493bf41360e2a14896667"
server_key_version: "encryptionkey@0"
I20260620 14:25:56.387637 3153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2611f7f43ff144129e3dcdd310ce89b0"
format_stamp: "Formatted at 2026-06-20 14:25:56 on dist-test-slave-pk3j"
server_key: "edbe90e830360a83034dc74baf586bd4"
server_key_iv: "6bf0c791c01493bf41360e2a14896667"
server_key_version: "encryptionkey@0"
I20260620 14:25:56.390833 3153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:25:56.392987 3168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:56.394167 3153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:56.394290 3153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "2611f7f43ff144129e3dcdd310ce89b0"
format_stamp: "Formatted at 2026-06-20 14:25:56 on dist-test-slave-pk3j"
server_key: "edbe90e830360a83034dc74baf586bd4"
server_key_iv: "6bf0c791c01493bf41360e2a14896667"
server_key_version: "encryptionkey@0"
I20260620 14:25:56.394388 3153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:56.409047 3153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:56.411387 3153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:56.411557 3153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:56.412040 3153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:56.412803 3153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:56.412855 3153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:56.412923 3153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:56.412955 3153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:56.421401 3153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41903
I20260620 14:25:56.421439 3281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41903 every 8 connection(s)
I20260620 14:25:56.422170 3153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:25:56.425345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 3153
I20260620 14:25:56.425458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:25:56.425711 392 external_mini_cluster.cc:1428] Setting key c794bac21a1c20a92967ed61857241fe
Jun 20 14:25:56 dist-test-slave-pk3j krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:56.434559 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:56.423725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42611 (local address 127.0.98.62:34079)
0620 14:25:56.423847 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:56.423851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:56.424566 (+ 715us) server_negotiation.cc:407] Connection header received
0620 14:25:56.426205 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:56.426209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:56.426264 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:56.426353 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:56.427791 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.428371 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.429434 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.429638 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.429715 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:56.431825 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:56.431828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:56.431830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:56.431850 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:56.432993 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:56.433459 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:56.433462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:56.433463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:56.433529 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:56.433776 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:56.433778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:56.433779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:56.433920 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:56.433985 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:56.434373 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 14:25:56.434454 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:25:56.435196 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:56.424045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34079 (local address 127.0.98.3:42611)
0620 14:25:56.424428 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:56.424472 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:56.425996 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:56.426575 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:56.426582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:56.427065 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:56.427655 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:56.427673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.428506 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.428509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:56.429335 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:56.429343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:56.429752 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:56.429755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:56.429878 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:56.430418 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:56.430433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:56.431694 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:56.433086 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:56.433090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:56.433099 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:56.433380 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:56.433604 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:56.433606 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:56.433607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:56.433718 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:56.433998 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:56.434002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:56.434181 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 14:25:56.434308 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:25:56.434509 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":104,"threads_started":1}
I20260620 14:25:56.436132 3282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34079
I20260620 14:25:56.436342 3282 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:56.436748 3282 heartbeater.cc:511] Master 127.0.98.62:34079 requested a full tablet report, sending...
I20260620 14:25:56.437642 2823 ts_manager.cc:194] Registered new tserver with Master: 2611f7f43ff144129e3dcdd310ce89b0 (127.0.98.3:41903)
I20260620 14:25:56.438174 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42611
I20260620 14:25:56.439822 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:25:56.451372 2862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:25:56.450105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.0.98.62:34079)
0620 14:25:56.450240 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:56.450244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:56.450389 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:25:56.450545 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:56.450548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:56.450592 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:56.450667 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:56.451279 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40202: BlockingRecv error: recv got EOF from 127.0.0.1:40202 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:25:56.452577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 2885
I20260620 14:25:56.457708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 3019
I20260620 14:25:56.462693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 3153
I20260620 14:25:56.467602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 2791
2026-06-20T14:25:56Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1965 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:25:56 dist-test-slave-pk3j krb5kdc[3294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:25:56 dist-test-slave-pk3j krb5kdc[3294](info): set up 2 sockets
Jun 20 14:25:56 dist-test-slave-pk3j krb5kdc[3294](info): commencing operation
krb5kdc: starting...
W20260620 14:25:57.469617 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:25:57 dist-test-slave-pk3j krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:25:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:25:57.576045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:57.676496 3307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:57.676743 3307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:57.676817 3307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:57.680622 3307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:57.680699 3307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:57.680737 3307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:57.680794 3307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:57.680832 3307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:57.685590 3307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:57.686473 3307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:57.687359 3307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:57.694907 3312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:57.694880 3313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:57.694902 3315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:57.695086 3307 server_base.cc:1034] running on GCE node
I20260620 14:25:57.695740 3307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:57.696693 3307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:57.697872 3307 hybrid_clock.cc:648] HybridClock initialized: now 1781965557697845 us; error 40 us; skew 500 ppm
Jun 20 14:25:57 dist-test-slave-pk3j krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:57.700275 3307 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:25:57.701061 3307 webserver.cc:460] Webserver started at http://127.0.98.62:43147/ using document root <none> and password file <none>
I20260620 14:25:57.701594 3307 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:57.701679 3307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:57.701884 3307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:57.703712 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "8c70828e679d43488d957905615d2ea1"
format_stamp: "Formatted at 2026-06-20 14:25:57 on dist-test-slave-pk3j"
server_key: "c3ea83a9b91f01d94020ebcc937792ae"
server_key_iv: "5389a33bc826290b5f5da2f4134d31ab"
server_key_version: "encryptionkey@0"
I20260620 14:25:57.704216 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c70828e679d43488d957905615d2ea1"
format_stamp: "Formatted at 2026-06-20 14:25:57 on dist-test-slave-pk3j"
server_key: "c3ea83a9b91f01d94020ebcc937792ae"
server_key_iv: "5389a33bc826290b5f5da2f4134d31ab"
server_key_version: "encryptionkey@0"
I20260620 14:25:57.707540 3307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:25:57.709630 3322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:57.710753 3307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:57.710909 3307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "8c70828e679d43488d957905615d2ea1"
format_stamp: "Formatted at 2026-06-20 14:25:57 on dist-test-slave-pk3j"
server_key: "c3ea83a9b91f01d94020ebcc937792ae"
server_key_iv: "5389a33bc826290b5f5da2f4134d31ab"
server_key_version: "encryptionkey@0"
I20260620 14:25:57.711016 3307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:57.723186 3307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:57.725677 3307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:57.725874 3307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:57.732116 3307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42949
I20260620 14:25:57.732376 3374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42949 every 8 connection(s)
I20260620 14:25:57.732995 3307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:57.735553 3375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:57.740537 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1: Bootstrap starting.
I20260620 14:25:57.741251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 3307
I20260620 14:25:57.741377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:57.741629 392 external_mini_cluster.cc:1428] Setting key e9c0a98393352bf36a0ac1e6b95db884
I20260620 14:25:57.743252 3375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:57.744086 3375 log.cc:824] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:57.746016 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1: No bootstrap required, opened a new log
I20260620 14:25:57.749763 3375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c70828e679d43488d957905615d2ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260620 14:25:57.749989 3375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:57.750077 3375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c70828e679d43488d957905615d2ea1, State: Initialized, Role: FOLLOWER
Jun 20 14:25:57 dist-test-slave-pk3j krb5kdc[3294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:57.750584 3375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c70828e679d43488d957905615d2ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260620 14:25:57.750737 3375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:25:57.750800 3375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:25:57.750895 3375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:57.751988 3375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c70828e679d43488d957905615d2ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260620 14:25:57.752349 3375 leader_election.cc:302] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c70828e679d43488d957905615d2ea1; no voters:
I20260620 14:25:57.752637 3375 leader_election.cc:288] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:25:57.752735 3380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:57.752961 3380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [term 1 LEADER]: Becoming Leader. State: Replica: 8c70828e679d43488d957905615d2ea1, State: Running, Role: LEADER
I20260620 14:25:57.753324 3380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c70828e679d43488d957905615d2ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } }
I20260620 14:25:57.753855 3375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:57.754781 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c70828e679d43488d957905615d2ea1. Latest consensus state: current_term: 1 leader_uuid: "8c70828e679d43488d957905615d2ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c70828e679d43488d957905615d2ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } } }
I20260620 14:25:57.754921 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:57.756381 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:57.742793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51162 (local address 127.0.98.62:42949)
0620 14:25:57.743234 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:57.743253 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:57.743294 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:25:57.744103 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:57.744125 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:57.744520 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:57.744829 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:57.745510 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:57.746889 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:57.747683 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:57.748345 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:57.748545 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:57.750994 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:57.751000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:57.751014 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:57.751042 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:57.752899 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:57.753411 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:57.753417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:57.753422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:57.753544 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:57.753846 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:57.753848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:57.753850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:57.754345 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:57.754557 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:57.755203 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 14:25:57.755484 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
I20260620 14:25:57.756731 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c70828e679d43488d957905615d2ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c70828e679d43488d957905615d2ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42949 } } }
I20260620 14:25:57.756836 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1 [sys.catalog]: This master's current role is: LEADER
W20260620 14:25:57.758492 3395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:25:57.758599 3395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:25:57.758699 3389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:57.759524 3389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:25:57.759788 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:25:57.764360 3389 catalog_manager.cc:1347] Generated new cluster ID: 86b33ff206a947aa88cb347769728ea6
I20260620 14:25:57.764429 3389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:25:57.777355 3389 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:57.778241 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:57.772870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.0.98.62:42949)
0620 14:25:57.772966 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:57.772969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:57.773221 (+ 252us) server_negotiation.cc:407] Connection header received
0620 14:25:57.773315 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:57.773318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:57.773362 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:57.773442 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:57.773949 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:57.774731 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:57.775832 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:57.776190 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:57.776300 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:57.777319 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:57.777323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:57.777326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:25:57.777791 (+ 465us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:57.777822 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:25:57.777840 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:57.777927 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:57.778058 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:25:57.778115 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":35}
I20260620 14:25:57.778352 3389 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:57.784022 3389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c70828e679d43488d957905615d2ea1: Generated new TSK 0
I20260620 14:25:57.784601 3389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:25:57.793162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 3307
2026-06-20T14:25:57Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1323 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:25:57 dist-test-slave-pk3j krb5kdc[3408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:25:57 dist-test-slave-pk3j krb5kdc[3408](info): set up 2 sockets
Jun 20 14:25:57 dist-test-slave-pk3j krb5kdc[3408](info): commencing operation
krb5kdc: starting...
W20260620 14:25:58.796968 392 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:25:58 dist-test-slave-pk3j krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:25:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:25:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:25:58.905493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35705
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:59.014354 3421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:59.014555 3421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:59.014591 3421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:59.018357 3421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:25:59.018399 3421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:59.018416 3421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:59.018431 3421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:25:59.018445 3421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:25:59.023108 3421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:59.023880 3421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:59.024660 3421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:59.032203 3426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:59.032192 3429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:59.032193 3427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:59.032235 3421 server_base.cc:1034] running on GCE node
I20260620 14:25:59.032884 3421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:59.033913 3421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:59.035094 3421 hybrid_clock.cc:648] HybridClock initialized: now 1781965559035079 us; error 31 us; skew 500 ppm
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:59.037657 3421 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:25:59.038420 3421 webserver.cc:460] Webserver started at http://127.0.98.62:34793/ using document root <none> and password file <none>
I20260620 14:25:59.038902 3421 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:59.038985 3421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:59.039197 3421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:59.041147 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "53c30f588dbb479cb1d0b9ce0c02ef17"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "e41c570305b0c199c7ac1c223793c1b2"
server_key_iv: "aadc9b752dc84621ddec7217b707eaa4"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.041702 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "53c30f588dbb479cb1d0b9ce0c02ef17"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "e41c570305b0c199c7ac1c223793c1b2"
server_key_iv: "aadc9b752dc84621ddec7217b707eaa4"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.044991 3421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:25:59.047173 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:59.048216 3421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:59.048349 3421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "53c30f588dbb479cb1d0b9ce0c02ef17"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "e41c570305b0c199c7ac1c223793c1b2"
server_key_iv: "aadc9b752dc84621ddec7217b707eaa4"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.048447 3421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:59.085103 3421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:59.087656 3421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:59.087841 3421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:59.093848 3421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35705
I20260620 14:25:59.094094 3488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35705 every 8 connection(s)
I20260620 14:25:59.094702 3421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:25:59.097110 3489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:25:59.100472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 3421
I20260620 14:25:59.100600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:25:59.100839 392 external_mini_cluster.cc:1428] Setting key ce367d292f9aebb3ed8636081db9eb98
I20260620 14:25:59.102170 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17: Bootstrap starting.
I20260620 14:25:59.104303 3489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17: Neither blocks nor log segments found. Creating new log.
I20260620 14:25:59.104997 3489 log.cc:824] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17: Log is configured to *not* fsync() on all Append() calls
I20260620 14:25:59.106665 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17: No bootstrap required, opened a new log
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:59.110078 3489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c30f588dbb479cb1d0b9ce0c02ef17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } }
I20260620 14:25:59.110302 3489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:25:59.110360 3489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c30f588dbb479cb1d0b9ce0c02ef17, State: Initialized, Role: FOLLOWER
I20260620 14:25:59.110742 3489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c30f588dbb479cb1d0b9ce0c02ef17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } }
I20260620 14:25:59.110872 3489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:25:59.110986 3489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:25:59.111105 3489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:25:59.111877 3489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c30f588dbb479cb1d0b9ce0c02ef17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } }
I20260620 14:25:59.112221 3489 leader_election.cc:302] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c30f588dbb479cb1d0b9ce0c02ef17; no voters:
I20260620 14:25:59.112562 3489 leader_election.cc:288] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:25:59.112654 3494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:25:59.112857 3494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [term 1 LEADER]: Becoming Leader. State: Replica: 53c30f588dbb479cb1d0b9ce0c02ef17, State: Running, Role: LEADER
I20260620 14:25:59.113292 3494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c30f588dbb479cb1d0b9ce0c02ef17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } }
I20260620 14:25:59.113708 3489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:25:59.114964 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:59.102119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.0.98.62:35705)
0620 14:25:59.102491 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:59.102506 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:59.102537 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:25:59.103108 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:59.103122 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:59.103403 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:59.103660 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:59.104541 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.105557 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.106427 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.106921 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.107090 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:59.109435 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:59.109441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:59.109453 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:59.109476 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:59.111396 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:59.111803 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:59.111808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:59.111813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:59.111919 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:59.112275 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:59.112277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:59.112278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:59.112757 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:59.113504 (+ 747us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:59.113633 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:25:59.114011 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":136,"threads_started":1}
I20260620 14:25:59.114977 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c30f588dbb479cb1d0b9ce0c02ef17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c30f588dbb479cb1d0b9ce0c02ef17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } } }
I20260620 14:25:59.115298 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [sys.catalog]: This master's current role is: LEADER
I20260620 14:25:59.115180 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c30f588dbb479cb1d0b9ce0c02ef17. Latest consensus state: current_term: 1 leader_uuid: "53c30f588dbb479cb1d0b9ce0c02ef17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c30f588dbb479cb1d0b9ce0c02ef17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35705 } } }
I20260620 14:25:59.115583 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17 [sys.catalog]: This master's current role is: LEADER
W20260620 14:25:59.118368 3509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:25:59.118429 3509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:25:59.118505 3505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:25:59.119143 3505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:25:59.123988 3505 catalog_manager.cc:1347] Generated new cluster ID: 5643c3bfd421463f9549e7dad853ff31
I20260620 14:25:59.124043 3505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:25:59.132622 3505 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:25:59.133704 3505 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:25:59.151955 3505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53c30f588dbb479cb1d0b9ce0c02ef17: Generated new TSK 0
I20260620 14:25:59.152671 3505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:25:59.181550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35705
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:59.310182 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:59.310374 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:59.310428 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:59.314208 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:59.314272 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:59.314352 3515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:25:59.319252 3515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:59.320092 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:59.320955 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:59.329375 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:59.329375 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:59.329437 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:59.329741 3515 server_base.cc:1034] running on GCE node
I20260620 14:25:59.330103 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:59.330626 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:59.331771 3515 hybrid_clock.cc:648] HybridClock initialized: now 1781965559331752 us; error 42 us; skew 500 ppm
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:59.334334 3515 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:25:59.335064 3515 webserver.cc:460] Webserver started at http://127.0.98.1:41239/ using document root <none> and password file <none>
I20260620 14:25:59.335542 3515 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:59.335624 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:59.335858 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:59.337673 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c8b4e80c9184309b6ff9aa0e6516989"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "ce3511e9d7238812aa05519ac6fb148a"
server_key_iv: "f4990dae4a487337098d8003742df7e5"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.338209 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c8b4e80c9184309b6ff9aa0e6516989"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "ce3511e9d7238812aa05519ac6fb148a"
server_key_iv: "f4990dae4a487337098d8003742df7e5"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.341862 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:25:59.344041 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:59.345053 3515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:59.345191 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "2c8b4e80c9184309b6ff9aa0e6516989"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "ce3511e9d7238812aa05519ac6fb148a"
server_key_iv: "f4990dae4a487337098d8003742df7e5"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.345350 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:59.354918 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:59.357921 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:59.358098 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:59.358568 3515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:59.359258 3515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:59.359326 3515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:59.359390 3515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:59.359433 3515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:59.367028 3515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40657
I20260620 14:25:59.367113 3643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40657 every 8 connection(s)
I20260620 14:25:59.367815 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:59.377406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 3515
I20260620 14:25:59.377513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:25:59.377791 392 external_mini_cluster.cc:1428] Setting key e41f3bc3fd09a238802f7bb0ecd13ea0
I20260620 14:25:59.380190 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:59.369353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51509 (local address 127.0.98.62:35705)
0620 14:25:59.369471 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:59.369475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:59.370179 (+ 704us) server_negotiation.cc:407] Connection header received
0620 14:25:59.371447 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:59.371450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:59.371506 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:59.371605 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:59.372608 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.373133 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.374253 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.374482 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.374592 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:59.376700 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:59.376703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:59.376705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:59.376728 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:59.378160 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:59.378632 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:59.378638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:59.378642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:59.378728 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:59.379100 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:59.379103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:59.379105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:59.379314 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:59.379419 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:59.379829 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 14:25:59.379962 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:25:59.380689 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:59.369667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35705 (local address 127.0.98.1:51509)
0620 14:25:59.370049 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:59.370085 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:59.371241 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:59.371765 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:59.371771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:59.372077 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:59.372470 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:59.372481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.373329 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.373332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:59.374130 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:59.374138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.374620 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.374624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:59.374788 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:59.375224 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:59.375238 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:59.376552 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:59.378264 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:59.378269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:59.378284 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:59.378516 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:59.378828 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:59.378833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:59.378835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:59.378999 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:59.379414 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:59.379418 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:59.379603 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 14:25:59.379747 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 14:25:59.379948 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":99,"threads_started":1}
I20260620 14:25:59.381814 3644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35705
I20260620 14:25:59.382025 3644 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:59.382438 3644 heartbeater.cc:511] Master 127.0.98.62:35705 requested a full tablet report, sending...
I20260620 14:25:59.383857 3452 ts_manager.cc:194] Registered new tserver with Master: 2c8b4e80c9184309b6ff9aa0e6516989 (127.0.98.1:40657)
I20260620 14:25:59.384871 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51509
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:25:59.401875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35705
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:59.500488 3649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:59.500695 3649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:59.500733 3649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:59.504456 3649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:59.504504 3649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:59.504557 3649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:25:59.509186 3649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:59.510004 3649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:59.510787 3649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:59.519258 3657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:59.519264 3654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:59.519264 3655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:59.519445 3649 server_base.cc:1034] running on GCE node
I20260620 14:25:59.519865 3649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:59.520452 3649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:59.521626 3649 hybrid_clock.cc:648] HybridClock initialized: now 1781965559521621 us; error 31 us; skew 500 ppm
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:59.523953 3649 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:25:59.524605 3649 webserver.cc:460] Webserver started at http://127.0.98.2:43979/ using document root <none> and password file <none>
I20260620 14:25:59.525017 3649 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:59.525074 3649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:59.525275 3649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:59.527024 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "dbee87d2f75b47b1ad9760598c53bb32"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "27864aed9ff722221c958e36f120dbec"
server_key_iv: "03a289c5703a7002917197b700a7fa4e"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.527498 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dbee87d2f75b47b1ad9760598c53bb32"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "27864aed9ff722221c958e36f120dbec"
server_key_iv: "03a289c5703a7002917197b700a7fa4e"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.530593 3649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:25:59.532621 3664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:59.533744 3649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:25:59.533878 3649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "dbee87d2f75b47b1ad9760598c53bb32"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "27864aed9ff722221c958e36f120dbec"
server_key_iv: "03a289c5703a7002917197b700a7fa4e"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.533972 3649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:59.555824 3649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:59.558554 3649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:59.558733 3649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:59.559232 3649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:59.559957 3649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:59.560039 3649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:59.560115 3649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:59.560150 3649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:59.568215 3649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39339
I20260620 14:25:59.568238 3777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39339 every 8 connection(s)
I20260620 14:25:59.568949 3649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:25:59.576633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 3649
I20260620 14:25:59.576786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:25:59.577042 392 external_mini_cluster.cc:1428] Setting key 0dac60c7b5dd080836bfa41cdb0af1c6
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:59.581718 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:59.570546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53087 (local address 127.0.98.62:35705)
0620 14:25:59.570683 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:59.570687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:59.571529 (+ 842us) server_negotiation.cc:407] Connection header received
0620 14:25:59.572792 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:59.572795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:59.572838 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:59.572909 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:59.573979 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.574505 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.575525 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.575723 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.575798 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:59.578248 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:59.578251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:59.578254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:59.578278 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:59.579859 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:59.580289 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:59.580295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:59.580297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:59.580393 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:59.580672 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:59.580676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:59.580678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:59.580889 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:59.580985 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:59.581446 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 14:25:59.581557 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:25:59.582468 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:59.570941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35705 (local address 127.0.98.2:53087)
0620 14:25:59.571383 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:59.571420 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:59.572615 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:59.573049 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:59.573055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:59.573468 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:59.573866 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:59.573878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.574622 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.574624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:59.575424 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:59.575432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.575838 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.575840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:59.575985 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:59.576489 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:59.576504 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:59.578095 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:59.579966 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:59.579971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:59.579980 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:59.580213 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:59.580478 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:59.580480 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:59.580481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:59.580589 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:59.580979 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:59.580983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:59.581174 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 14:25:59.581395 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 14:25:59.581659 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260620 14:25:59.583528 3778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35705
I20260620 14:25:59.583755 3778 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:59.584173 3778 heartbeater.cc:511] Master 127.0.98.62:35705 requested a full tablet report, sending...
I20260620 14:25:59.585096 3452 ts_manager.cc:194] Registered new tserver with Master: dbee87d2f75b47b1ad9760598c53bb32 (127.0.98.2:39339)
I20260620 14:25:59.585886 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53087
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:25:59.601572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35705
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:25:59.702781 3783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:25:59.702978 3783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:25:59.703014 3783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:25:59.706854 3783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:25:59.706902 3783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:25:59.707001 3783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:25:59.711707 3783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:25:59.712539 3783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:25:59.713464 3783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:25:59.722150 3791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:59.722172 3789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:25:59.722151 3788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:25:59.722241 3783 server_base.cc:1034] running on GCE node
I20260620 14:25:59.722769 3783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:25:59.723304 3783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:25:59.724474 3783 hybrid_clock.cc:648] HybridClock initialized: now 1781965559724466 us; error 25 us; skew 500 ppm
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:25:59.726963 3783 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:25:59.727798 3783 webserver.cc:460] Webserver started at http://127.0.98.3:39165/ using document root <none> and password file <none>
I20260620 14:25:59.728303 3783 fs_manager.cc:359] Metadata directory not provided
I20260620 14:25:59.728387 3783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:25:59.728598 3783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:25:59.730422 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "9a0c914a6fc74eebaa33543acc4bfd5c"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "45551d90a8e1668e6d36703423acc444"
server_key_iv: "1a7f77eef066d41ccd6756455d60b239"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.730893 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9a0c914a6fc74eebaa33543acc4bfd5c"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "45551d90a8e1668e6d36703423acc444"
server_key_iv: "1a7f77eef066d41ccd6756455d60b239"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.734117 3783 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:25:59.736253 3798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:59.737445 3783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:25:59.737551 3783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "9a0c914a6fc74eebaa33543acc4bfd5c"
format_stamp: "Formatted at 2026-06-20 14:25:59 on dist-test-slave-pk3j"
server_key: "45551d90a8e1668e6d36703423acc444"
server_key_iv: "1a7f77eef066d41ccd6756455d60b239"
server_key_version: "encryptionkey@0"
I20260620 14:25:59.737655 3783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:25:59.758111 3783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:25:59.760483 3783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:25:59.760660 3783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:25:59.761147 3783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:25:59.761871 3783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:25:59.761940 3783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:59.762001 3783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:25:59.762042 3783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:25:59.770213 3783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35009
I20260620 14:25:59.770234 3911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35009 every 8 connection(s)
I20260620 14:25:59.770941 3783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:25:59.776118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 3783
I20260620 14:25:59.776199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:25:59.776461 392 external_mini_cluster.cc:1428] Setting key 6f7f37ba82cb4ca4471c5a1e0986ee6e
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:59.783262 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:59.772545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33569 (local address 127.0.98.62:35705)
0620 14:25:59.772699 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:59.772705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:59.773445 (+ 740us) server_negotiation.cc:407] Connection header received
0620 14:25:59.774663 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:59.774666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:59.774709 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:59.774783 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:59.775754 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.776355 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.777748 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.777990 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.778068 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:59.780329 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:59.780332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:59.780334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:59.780354 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:59.781726 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:59.782149 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:59.782151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:59.782153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:59.782219 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:59.782483 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:59.782485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:59.782486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:59.782638 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:59.782707 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:59.783071 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 14:25:59.783154 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:25:59.783897 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:59.772892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35705 (local address 127.0.98.3:33569)
0620 14:25:59.773322 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 14:25:59.773359 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:25:59.774468 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:25:59.774925 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:25:59.774931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:25:59.775247 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:25:59.775639 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:59.775650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.776472 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.776479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:59.777623 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:25:59.777632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.778142 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.778146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:25:59.778325 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:59.778847 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:25:59.778861 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:25:59.780193 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:25:59.781815 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:59.781819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:59.781829 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:59.782044 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:59.782301 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:25:59.782303 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:25:59.782305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:25:59.782413 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:25:59.782716 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:25:59.782720 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:25:59.782895 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 14:25:59.783029 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 14:25:59.783231 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
I20260620 14:25:59.784811 3912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35705
I20260620 14:25:59.785009 3912 heartbeater.cc:464] Registering TS with master...
I20260620 14:25:59.785449 3912 heartbeater.cc:511] Master 127.0.98.62:35705 requested a full tablet report, sending...
I20260620 14:25:59.786329 3452 ts_manager.cc:194] Registered new tserver with Master: 9a0c914a6fc74eebaa33543acc4bfd5c (127.0.98.3:35009)
I20260620 14:25:59.786849 3452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33569
I20260620 14:25:59.789981 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965559, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965559, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:25:59.812381 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:25:59.805258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.0.98.62:35705)
0620 14:25:59.805362 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:25:59.805365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:25:59.805546 (+ 181us) server_negotiation.cc:407] Connection header received
0620 14:25:59.805638 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:25:59.805640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:25:59.805673 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:25:59.805738 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:25:59.806391 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.806856 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.807614 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:25:59.807805 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:25:59.807879 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:25:59.809974 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:25:59.809977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:25:59.809978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:25:59.809998 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:25:59.811137 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:59.811457 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:59.811458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:59.811459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:59.811519 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:25:59.811713 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:25:59.811714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:25:59.811715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:25:59.811878 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:25:59.811967 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:25:59.812177 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:25:59.812279 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 14:25:59.812685 3452 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38176
I20260620 14:25:59.813493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 3515
I20260620 14:25:59.819129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 3649
I20260620 14:25:59.824218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 3783
I20260620 14:25:59.829135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 3421
2026-06-20T14:25:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2037 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3924](info): set up 2 sockets
Jun 20 14:25:59 dist-test-slave-pk3j krb5kdc[3924](info): commencing operation
krb5kdc: starting...
W20260620 14:26:00.828965 392 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:26:00 dist-test-slave-pk3j krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:26:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:26:00.934299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43121
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:01.035907 3937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:01.036187 3937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:01.036257 3937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:01.040133 3937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:01.040210 3937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:01.040246 3937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:01.040300 3937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:01.040334 3937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:01.045248 3937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:01.046140 3937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:01.047024 3937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:01.054723 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:01.054801 3937 server_base.cc:1034] running on GCE node
W20260620 14:26:01.054736 3942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:01.054736 3943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:01.055490 3937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:01.056375 3937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:01.057523 3937 hybrid_clock.cc:648] HybridClock initialized: now 1781965561057506 us; error 30 us; skew 500 ppm
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:01.060055 3937 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:26:01.060850 3937 webserver.cc:460] Webserver started at http://127.0.98.62:33783/ using document root <none> and password file <none>
I20260620 14:26:01.061363 3937 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:01.061429 3937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:01.061574 3937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:01.063320 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "15fb4b4269ed4a40810bc3e4f6349872"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "a3e4014727e31a7f10f6f74cdb977d67"
server_key_iv: "fec3d61b0c3796969999b43267287a6e"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.063781 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "15fb4b4269ed4a40810bc3e4f6349872"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "a3e4014727e31a7f10f6f74cdb977d67"
server_key_iv: "fec3d61b0c3796969999b43267287a6e"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.067065 3937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:01.069150 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:01.070297 3937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:01.070428 3937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "15fb4b4269ed4a40810bc3e4f6349872"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "a3e4014727e31a7f10f6f74cdb977d67"
server_key_iv: "fec3d61b0c3796969999b43267287a6e"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.070528 3937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:01.080729 3937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:01.083135 3937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:01.083318 3937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:01.089630 3937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43121
I20260620 14:26:01.089644 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43121 every 8 connection(s)
I20260620 14:26:01.090416 3937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:01.092789 4005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:01.097404 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872: Bootstrap starting.
I20260620 14:26:01.098901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 3937
I20260620 14:26:01.099037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:01.099284 392 external_mini_cluster.cc:1428] Setting key 89ce2b6d0dc930553adcdd66f1bd574d
I20260620 14:26:01.099521 4005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:01.100258 4005 log.cc:824] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:01.101866 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872: No bootstrap required, opened a new log
I20260620 14:26:01.104820 4005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fb4b4269ed4a40810bc3e4f6349872" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43121 } }
I20260620 14:26:01.105036 4005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:01.105109 4005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15fb4b4269ed4a40810bc3e4f6349872, State: Initialized, Role: FOLLOWER
I20260620 14:26:01.105535 4005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fb4b4269ed4a40810bc3e4f6349872" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43121 } }
I20260620 14:26:01.105685 4005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:01.105767 4005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:01.105877 4005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:01.106643 4005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fb4b4269ed4a40810bc3e4f6349872" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43121 } }
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:01.106966 4005 leader_election.cc:302] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15fb4b4269ed4a40810bc3e4f6349872; no voters:
I20260620 14:26:01.107292 4005 leader_election.cc:288] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:01.107349 4010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:01.107607 4010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [term 1 LEADER]: Becoming Leader. State: Replica: 15fb4b4269ed4a40810bc3e4f6349872, State: Running, Role: LEADER
I20260620 14:26:01.108165 4010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fb4b4269ed4a40810bc3e4f6349872" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43121 } }
I20260620 14:26:01.108508 4005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:01.109843 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15fb4b4269ed4a40810bc3e4f6349872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fb4b4269ed4a40810bc3e4f6349872" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43121 } } }
I20260620 14:26:01.110030 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:01.109855 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15fb4b4269ed4a40810bc3e4f6349872. Latest consensus state: current_term: 1 leader_uuid: "15fb4b4269ed4a40810bc3e4f6349872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15fb4b4269ed4a40810bc3e4f6349872" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43121 } } }
I20260620 14:26:01.110387 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:01.110437 4022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:01.112608 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:01.100247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.0.98.62:43121)
0620 14:26:01.100656 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:01.100669 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:01.100699 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:26:01.101297 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:01.101313 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:01.101590 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:01.101887 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:01.102637 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.103490 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.104311 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.104740 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.104901 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:01.107541 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:01.107546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:01.107557 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:01.107581 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:01.109805 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.110278 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.110287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.110294 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.110423 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.110747 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.110752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.110755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.111128 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:01.111333 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:01.111600 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:26:01.111860 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":146,"threads_started":1}
I20260620 14:26:01.113248 4022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:01.118239 4022 catalog_manager.cc:1347] Generated new cluster ID: e3c1a447c89f4b68a38529478d34372d
I20260620 14:26:01.118300 4022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:01.128528 4022 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:01.129350 4022 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:01.139802 4022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15fb4b4269ed4a40810bc3e4f6349872: Generated new TSK 0
I20260620 14:26:01.140357 4022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:26:01.172786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43121
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:01.277995 4031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:01.278209 4031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:01.278247 4031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:01.282055 4031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:01.282099 4031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:01.282197 4031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:01.287057 4031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:01.287914 4031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:01.288794 4031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:01.297701 4036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:01.297737 4039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:01.297732 4037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:01.297854 4031 server_base.cc:1034] running on GCE node
I20260620 14:26:01.298324 4031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:01.298892 4031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:01.300046 4031 hybrid_clock.cc:648] HybridClock initialized: now 1781965561300020 us; error 39 us; skew 500 ppm
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:01.302605 4031 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:26:01.303442 4031 webserver.cc:460] Webserver started at http://127.0.98.1:42509/ using document root <none> and password file <none>
I20260620 14:26:01.303942 4031 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:01.304005 4031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:01.304196 4031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:01.306103 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "688cdc2f256240e4b02faac3ab750d50"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "70975ec53e5f5ee632b7473d808375da"
server_key_iv: "4e5377abb5443b36d58bdb550f68b0c0"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.306635 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "688cdc2f256240e4b02faac3ab750d50"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "70975ec53e5f5ee632b7473d808375da"
server_key_iv: "4e5377abb5443b36d58bdb550f68b0c0"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.310312 4031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:26:01.312461 4046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:01.313747 4031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:26:01.313868 4031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "688cdc2f256240e4b02faac3ab750d50"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "70975ec53e5f5ee632b7473d808375da"
server_key_iv: "4e5377abb5443b36d58bdb550f68b0c0"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.313966 4031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:01.334717 4031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:01.337471 4031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:01.337657 4031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:01.338210 4031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:01.338938 4031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:01.338992 4031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:01.339030 4031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:01.339046 4031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:01.347754 4031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41461
I20260620 14:26:01.347759 4159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41461 every 8 connection(s)
I20260620 14:26:01.348659 4031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:01.358125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 4031
I20260620 14:26:01.358263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:26:01.358527 392 external_mini_cluster.cc:1428] Setting key 5abd74ef147574cc189d6d17aaa95ff0
I20260620 14:26:01.362037 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:01.350552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50845 (local address 127.0.98.62:43121)
0620 14:26:01.350678 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:01.350682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:01.351585 (+ 903us) server_negotiation.cc:407] Connection header received
0620 14:26:01.352974 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:01.352977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:01.353030 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:01.353111 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:01.354249 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.354796 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.355884 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.356137 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.356251 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:01.358677 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:01.358688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:01.358690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:01.358713 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:01.360123 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.360605 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.360609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.360611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.360685 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.361010 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.361013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.361015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.361179 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:01.361310 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:01.361767 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 14:26:01.361879 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:26:01.362721 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:01.350992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43121 (local address 127.0.98.1:50845)
0620 14:26:01.351442 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:01.351482 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:01.352763 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:01.353308 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:01.353314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:01.353640 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:01.354105 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:01.354118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.354923 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.354927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:01.355763 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:01.355771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.356280 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.356284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:01.356451 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:01.356988 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:01.357003 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:01.358491 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:01.360251 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:01.360256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:01.360268 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:01.360503 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:01.360776 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:01.360780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:01.360782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:01.360911 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:01.361318 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:01.361326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:01.361543 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 14:26:01.361715 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:26:01.361993 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260620 14:26:01.363790 4160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43121
I20260620 14:26:01.364022 4160 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:01.364492 4160 heartbeater.cc:511] Master 127.0.98.62:43121 requested a full tablet report, sending...
I20260620 14:26:01.366214 3969 ts_manager.cc:194] Registered new tserver with Master: 688cdc2f256240e4b02faac3ab750d50 (127.0.98.1:41461)
I20260620 14:26:01.367771 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50845
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:26:01.383834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43121
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:01.486791 4165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:01.487004 4165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:01.487042 4165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:01.490825 4165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:01.490875 4165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:01.490973 4165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:01.495944 4165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:01.496801 4165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:01.497758 4165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:01.506253 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:01.506299 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:01.506328 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:01.506453 4165 server_base.cc:1034] running on GCE node
I20260620 14:26:01.506886 4165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:01.507470 4165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:01.508651 4165 hybrid_clock.cc:648] HybridClock initialized: now 1781965561508639 us; error 27 us; skew 500 ppm
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:01.511312 4165 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:26:01.512044 4165 webserver.cc:460] Webserver started at http://127.0.98.2:41505/ using document root <none> and password file <none>
I20260620 14:26:01.512487 4165 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:01.512559 4165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:01.512698 4165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:01.514465 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "21c2f2be64ea4331a10e1d7d4ef15428"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "f078571d312d13bad949d3c647a4194f"
server_key_iv: "b8f1ecc9055cad104d48c26d887ba745"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.514935 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "21c2f2be64ea4331a10e1d7d4ef15428"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "f078571d312d13bad949d3c647a4194f"
server_key_iv: "b8f1ecc9055cad104d48c26d887ba745"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.518355 4165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 14:26:01.520345 4180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:01.521418 4165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:26:01.521502 4165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "21c2f2be64ea4331a10e1d7d4ef15428"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "f078571d312d13bad949d3c647a4194f"
server_key_iv: "b8f1ecc9055cad104d48c26d887ba745"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.521572 4165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:01.539469 4165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:01.541831 4165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:01.541956 4165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:01.542402 4165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:01.543073 4165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:01.543116 4165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:01.543149 4165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:01.543169 4165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:01.551091 4165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40867
I20260620 14:26:01.551177 4293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40867 every 8 connection(s)
I20260620 14:26:01.551873 4165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:01.558719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 4165
I20260620 14:26:01.558863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:26:01.559131 392 external_mini_cluster.cc:1428] Setting key da527d371b073990f363f9ec6d8e3365
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:01.564699 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:01.553503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58487 (local address 127.0.98.62:43121)
0620 14:26:01.553619 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:01.553623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:01.554267 (+ 644us) server_negotiation.cc:407] Connection header received
0620 14:26:01.555525 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:01.555529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:01.555587 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:01.555688 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:01.556716 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.557308 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.558510 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.558724 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.558835 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:01.561359 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:01.561363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:01.561366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:01.561389 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:01.562764 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.563241 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.563246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.563248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.563323 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.563645 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.563649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.563651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.563811 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:01.563932 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:01.564505 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 14:26:01.564591 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:26:01.565784 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:01.553756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43121 (local address 127.0.98.2:58487)
0620 14:26:01.554135 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:01.554170 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:01.555332 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:01.555840 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:01.555846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:01.556173 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:01.556580 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:01.556591 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.557430 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.557434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:01.558339 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:01.558348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.558868 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.558872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:01.559046 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:01.559702 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:01.559728 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:01.561168 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:01.562893 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:01.562898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:01.562910 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:01.563144 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:01.563412 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:01.563416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:01.563418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:01.563552 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:01.563955 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:01.563964 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:01.564260 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 14:26:01.564458 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 14:26:01.564782 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":102,"threads_started":1}
I20260620 14:26:01.566896 4294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43121
I20260620 14:26:01.567139 4294 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:01.567557 4294 heartbeater.cc:511] Master 127.0.98.62:43121 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:26:01.568498 3969 ts_manager.cc:194] Registered new tserver with Master: 21c2f2be64ea4331a10e1d7d4ef15428 (127.0.98.2:40867)
I20260620 14:26:01.569021 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:26:01.583189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43121
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:01.683187 4299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:01.683410 4299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:01.683480 4299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:01.687335 4299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:01.687409 4299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:01.687496 4299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:01.692245 4299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:01.693118 4299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:01.694001 4299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:01.702281 4305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:01.702288 4307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:01.702267 4304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:01.702507 4299 server_base.cc:1034] running on GCE node
I20260620 14:26:01.703259 4299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:01.703799 4299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:01.704972 4299 hybrid_clock.cc:648] HybridClock initialized: now 1781965561704948 us; error 45 us; skew 500 ppm
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:01.707607 4299 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:26:01.708423 4299 webserver.cc:460] Webserver started at http://127.0.98.3:36811/ using document root <none> and password file <none>
I20260620 14:26:01.708963 4299 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:01.709044 4299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:01.709316 4299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:01.711122 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "f54466242b2444a8871c21618de6c314"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "395a89b7010a911e0badbfea07b29f43"
server_key_iv: "1fa839516c34ff91fceab01ec5795486"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.711589 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f54466242b2444a8871c21618de6c314"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "395a89b7010a911e0badbfea07b29f43"
server_key_iv: "1fa839516c34ff91fceab01ec5795486"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.714819 4299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:01.716980 4314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:01.718291 4299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:01.718376 4299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "f54466242b2444a8871c21618de6c314"
format_stamp: "Formatted at 2026-06-20 14:26:01 on dist-test-slave-pk3j"
server_key: "395a89b7010a911e0badbfea07b29f43"
server_key_iv: "1fa839516c34ff91fceab01ec5795486"
server_key_version: "encryptionkey@0"
I20260620 14:26:01.718448 4299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:01.729202 4299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:01.731526 4299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:01.731652 4299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:01.732092 4299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:01.732730 4299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:01.732774 4299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:01.732826 4299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:01.732847 4299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:01.740872 4299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33485
I20260620 14:26:01.740896 4427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33485 every 8 connection(s)
I20260620 14:26:01.741693 4299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:01.747794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 4299
I20260620 14:26:01.747876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:26:01.748129 392 external_mini_cluster.cc:1428] Setting key 1370a39d2b20bb34218795c02d98b569
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:01.754211 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:01.743240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41717 (local address 127.0.98.62:43121)
0620 14:26:01.743362 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:01.743366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:01.744018 (+ 652us) server_negotiation.cc:407] Connection header received
0620 14:26:01.745405 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:01.745409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:01.745468 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:01.745571 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:01.746602 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.747142 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.748342 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.748668 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.748787 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:01.751106 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:01.751109 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:01.751111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:01.751132 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:01.752566 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.752966 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.752968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.752969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.753058 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.753400 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.753402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.753403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.753565 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:01.753637 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:01.754026 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 14:26:01.754107 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:26:01.754876 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:01.743493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43121 (local address 127.0.98.3:41717)
0620 14:26:01.743862 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:01.743898 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:01.745149 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:01.745724 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:01.745731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:01.746044 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:01.746446 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:01.746457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.747279 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.747283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:01.748224 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:01.748234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.748841 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.748847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:01.749031 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:01.749596 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:01.749611 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:01.750970 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:01.752655 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:01.752661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:01.752669 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:01.752885 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:01.753133 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:01.753135 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:01.753136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:01.753324 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:01.753646 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:01.753650 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:01.753829 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 14:26:01.753957 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:26:01.754164 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260620 14:26:01.755770 4428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43121
I20260620 14:26:01.756002 4428 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:01.756412 4428 heartbeater.cc:511] Master 127.0.98.62:43121 requested a full tablet report, sending...
I20260620 14:26:01.757320 3969 ts_manager.cc:194] Registered new tserver with Master: f54466242b2444a8871c21618de6c314 (127.0.98.3:33485)
I20260620 14:26:01.757793 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41717
I20260620 14:26:01.761463 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:26:01.784345 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:01.776054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.0.98.1:41461)
0620 14:26:01.776341 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:01.776351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:01.776370 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:26:01.776411 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:01.776415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:01.776564 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:01.776680 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:01.777321 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.778043 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.778803 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.779121 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.779201 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:01.781518 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:01.781527 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:01.781537 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:01.781559 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:01.782871 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.783309 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.783314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.783315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.783384 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.783599 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.783601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.783602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.783814 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:01.783947 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:01.784100 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 14:26:01.784207 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:01.794719 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:01.786951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.0.98.62:43121)
0620 14:26:01.787084 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:01.787091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:01.787154 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:26:01.787320 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:01.787323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:01.787366 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:01.787448 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:01.787993 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.788471 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.789225 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:01.789429 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:01.789534 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:01.791759 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:01.791765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:01.791772 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:01.791792 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:01.793121 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.793555 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.793559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.793560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.793634 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:01.793898 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:01.793903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:01.793904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:01.794098 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:01.794231 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:01.794449 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:26:01.794556 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 14:26:01.795097 3969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40126
I20260620 14:26:01.795680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 4031
I20260620 14:26:01.800909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 4165
I20260620 14:26:01.805941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 4299
I20260620 14:26:01.810824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 3937
2026-06-20T14:26:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1981 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[4440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[4440](info): set up 2 sockets
Jun 20 14:26:01 dist-test-slave-pk3j krb5kdc[4440](info): commencing operation
krb5kdc: starting...
W20260620 14:26:02.809134 392 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:26:02 dist-test-slave-pk3j krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:26:02.917407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37995
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:03.016716 4453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:03.016973 4453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:03.017059 4453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:03.020761 4453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:03.020839 4453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:03.020875 4453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:03.020928 4453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:03.020961 4453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:03.025594 4453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:03.026448 4453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:03.027333 4453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:03.034578 4461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:03.034628 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:03.034694 4453 server_base.cc:1034] running on GCE node
W20260620 14:26:03.034636 4458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:03.035279 4453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:03.036262 4453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:03.037429 4453 hybrid_clock.cc:648] HybridClock initialized: now 1781965563037422 us; error 30 us; skew 500 ppm
Jun 20 14:26:03 dist-test-slave-pk3j krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:03.039844 4453 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:26:03.040481 4453 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:03.040568 4453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:03.040767 4453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:03.042591 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "a48dea6732d04eaba9539956a72b834e"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "537d6c055ce68cda3beafc9c2e9b3b39"
server_key_iv: "62f73cbc04d37202774913068da5d3ef"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.043099 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "a48dea6732d04eaba9539956a72b834e"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "537d6c055ce68cda3beafc9c2e9b3b39"
server_key_iv: "62f73cbc04d37202774913068da5d3ef"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.046187 4453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 14:26:03.048220 4467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:03.049255 4453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:26:03.049376 4453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "a48dea6732d04eaba9539956a72b834e"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "537d6c055ce68cda3beafc9c2e9b3b39"
server_key_iv: "62f73cbc04d37202774913068da5d3ef"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.049474 4453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:03.066205 4453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:03.068514 4453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:03.068706 4453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:03.074760 4453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37995
I20260620 14:26:03.074987 4519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37995 every 8 connection(s)
I20260620 14:26:03.075314 4453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:03.077725 4520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:03.082240 4520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e: Bootstrap starting.
I20260620 14:26:03.082252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 4453
I20260620 14:26:03.082396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:03.082659 392 external_mini_cluster.cc:1428] Setting key 7957462f76cca6f011c0d6b604b11113
I20260620 14:26:03.084582 4520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:03.085322 4520 log.cc:824] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:03.086966 4520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e: No bootstrap required, opened a new log
I20260620 14:26:03.090206 4520 raft_consensus.cc:348] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48dea6732d04eaba9539956a72b834e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } }
I20260620 14:26:03.090452 4520 raft_consensus.cc:374] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:03.090520 4520 raft_consensus.cc:729] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48dea6732d04eaba9539956a72b834e, State: Initialized, Role: FOLLOWER
Jun 20 14:26:03 dist-test-slave-pk3j krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:03.090890 4520 consensus_queue.cc:260] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48dea6732d04eaba9539956a72b834e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } }
I20260620 14:26:03.091024 4520 raft_consensus.cc:388] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:03.091086 4520 raft_consensus.cc:482] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:03.091200 4520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:03.092223 4520 raft_consensus.cc:504] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48dea6732d04eaba9539956a72b834e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } }
I20260620 14:26:03.092536 4520 leader_election.cc:302] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a48dea6732d04eaba9539956a72b834e; no voters:
I20260620 14:26:03.092824 4520 leader_election.cc:288] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:03.092882 4525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:03.093057 4525 raft_consensus.cc:686] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [term 1 LEADER]: Becoming Leader. State: Replica: a48dea6732d04eaba9539956a72b834e, State: Running, Role: LEADER
I20260620 14:26:03.093379 4525 consensus_queue.cc:237] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48dea6732d04eaba9539956a72b834e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } }
I20260620 14:26:03.093933 4520 sys_catalog.cc:564] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:03.094821 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a48dea6732d04eaba9539956a72b834e. Latest consensus state: current_term: 1 leader_uuid: "a48dea6732d04eaba9539956a72b834e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48dea6732d04eaba9539956a72b834e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } }
I20260620 14:26:03.095222 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:03.095122 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a48dea6732d04eaba9539956a72b834e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48dea6732d04eaba9539956a72b834e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37995 } } }
I20260620 14:26:03.095530 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:03.096604 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.083802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.0.98.62:37995)
0620 14:26:03.084147 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.084159 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.084206 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:26:03.084852 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.084867 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.085157 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.085510 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:03.086265 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.087258 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.088020 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.088428 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.088584 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.091080 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:03.091088 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:03.091109 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:03.091138 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:03.092856 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:03.093401 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:03.093405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:03.093410 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:03.093526 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:03.093786 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:03.093791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:03.093793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:03.094196 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:03.094408 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.095193 (+ 785us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.095475 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":112,"threads_started":1}
W20260620 14:26:03.099045 4540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:26:03.099121 4540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:26:03.099227 4536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:03.099865 4536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:03.104745 4536 catalog_manager.cc:1347] Generated new cluster ID: 2445cd15714f4ce0a55869bd7500b7df
I20260620 14:26:03.104799 4536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:03.127532 4536 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:03.128368 4536 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:03.138823 4536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e: Generated new TSK 0
I20260620 14:26:03.139407 4536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:26:03.163699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:03.270164 4546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:03.270380 4546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:03.270419 4546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:03.274243 4546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:03.274289 4546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:03.274343 4546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:03.279213 4546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:03.280040 4546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:03.280881 4546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:03.289302 4552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:03.289354 4554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:03.289302 4551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:03.289474 4546 server_base.cc:1034] running on GCE node
I20260620 14:26:03.289932 4546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:03.290510 4546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:03.291692 4546 hybrid_clock.cc:648] HybridClock initialized: now 1781965563291680 us; error 24 us; skew 500 ppm
Jun 20 14:26:03 dist-test-slave-pk3j krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:03.294317 4546 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:26:03.295090 4546 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:03.295176 4546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:03.295367 4546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:03.297356 4546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ad991c0fcfe490d8cd6a69fd268621c"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "8d58f08621b3d46610edf3622d6d7667"
server_key_iv: "d136f0225bdb667af9101a6e83997972"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.297885 4546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ad991c0fcfe490d8cd6a69fd268621c"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "8d58f08621b3d46610edf3622d6d7667"
server_key_iv: "d136f0225bdb667af9101a6e83997972"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.301239 4546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:26:03.303515 4560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:03.304654 4546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:03.304775 4546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "4ad991c0fcfe490d8cd6a69fd268621c"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "8d58f08621b3d46610edf3622d6d7667"
server_key_iv: "d136f0225bdb667af9101a6e83997972"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.304873 4546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:03.315164 4546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:03.317828 4546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:03.318027 4546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:03.318532 4546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:03.319284 4546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:03.319350 4546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:03.319410 4546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:03.319455 4546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:03.327695 4546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35575
I20260620 14:26:03.327714 4673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35575 every 8 connection(s)
I20260620 14:26:03.328203 4546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:26:03.329070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 4546
I20260620 14:26:03.329192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:26:03.329526 392 external_mini_cluster.cc:1428] Setting key a772daac0b99fe4c3ac7d94807475c4d
Jun 20 14:26:03 dist-test-slave-pk3j krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:26:03.342554 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.329972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40677 (local address 127.0.98.62:37995)
0620 14:26:03.330118 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.330122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.330999 (+ 877us) server_negotiation.cc:407] Connection header received
0620 14:26:03.332523 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.332526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.332578 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.332659 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:03.333969 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.334613 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.336409 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.336644 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.336723 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.339311 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:03.339314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:03.339316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:03.339339 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:03.340511 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:03.341056 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:03.341062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:03.341065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:03.341157 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:03.341522 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:03.341527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:03.341528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:03.341702 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:03.341809 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.342253 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.342389 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:26:03.343271 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.330325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37995 (local address 127.0.98.1:40677)
0620 14:26:03.330856 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:03.330910 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:03.332333 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:03.332824 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:03.332830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:03.333145 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:03.333809 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.333829 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.334750 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.334756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.336275 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.336290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.336797 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.336804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.337007 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.337579 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:03.337602 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:03.339153 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:03.340612 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:03.340616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:03.340626 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:03.340952 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:03.341294 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:03.341297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:03.341299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:03.341426 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:03.341805 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:03.341811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:03.342030 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 14:26:03.342202 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:26:03.342508 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":130,"threads_started":1}
I20260620 14:26:03.344488 4674 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37995
I20260620 14:26:03.344730 4674 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:03.345247 4674 heartbeater.cc:511] Master 127.0.98.62:37995 requested a full tablet report, sending...
I20260620 14:26:03.346719 4484 ts_manager.cc:194] Registered new tserver with Master: 4ad991c0fcfe490d8cd6a69fd268621c (127.0.98.1:35575)
I20260620 14:26:03.347784 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:26:03.353971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:03.457479 4679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:03.457682 4679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:03.457746 4679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:03.461571 4679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:03.461616 4679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:03.461717 4679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:03.466322 4679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:03.467265 4679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:03.468170 4679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:03.476608 4685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:03.476686 4679 server_base.cc:1034] running on GCE node
W20260620 14:26:03.476644 4687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:03.476652 4684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:03.477270 4679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:03.477762 4679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:03.478965 4679 hybrid_clock.cc:648] HybridClock initialized: now 1781965563478941 us; error 42 us; skew 500 ppm
Jun 20 14:26:03 dist-test-slave-pk3j krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:03.481446 4679 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:26:03.482118 4679 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:03.482204 4679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:03.482452 4679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:03.484229 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "29e77d40b84d4c35b19fd00644cf33fb"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "550e2231e9ecf45335e32ad82bc3344f"
server_key_iv: "4596c11bec44e977c1aead698719a6cc"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.484774 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "29e77d40b84d4c35b19fd00644cf33fb"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "550e2231e9ecf45335e32ad82bc3344f"
server_key_iv: "4596c11bec44e977c1aead698719a6cc"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.488122 4679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:26:03.490306 4693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:03.491293 4679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:03.491434 4679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "29e77d40b84d4c35b19fd00644cf33fb"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "550e2231e9ecf45335e32ad82bc3344f"
server_key_iv: "4596c11bec44e977c1aead698719a6cc"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.491533 4679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:03.507349 4679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:03.510093 4679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:03.510262 4679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:03.510742 4679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:03.511447 4679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:03.511514 4679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:03.511610 4679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:03.511662 4679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:03.519621 4679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34153
I20260620 14:26:03.519642 4806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34153 every 8 connection(s)
I20260620 14:26:03.520126 4679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:26:03 dist-test-slave-pk3j krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:03.529277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 4679
I20260620 14:26:03.529395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:26:03.529729 392 external_mini_cluster.cc:1428] Setting key 7f24081bc3c6de791fc900f201e91e65
I20260620 14:26:03.533004 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.521696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47769 (local address 127.0.98.62:37995)
0620 14:26:03.521818 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.521827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.522644 (+ 817us) server_negotiation.cc:407] Connection header received
0620 14:26:03.524107 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.524110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.524166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.524242 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:03.525293 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.525805 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.526914 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.527142 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.527265 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.529557 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:03.529562 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:03.529565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:03.529597 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:03.531008 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:03.531461 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:03.531467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:03.531472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:03.531559 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:03.531858 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:03.531864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:03.531869 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:03.532100 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:03.532200 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.532739 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.532848 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:26:03.533771 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.522084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37995 (local address 127.0.98.2:47769)
0620 14:26:03.522484 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:03.522522 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:03.523901 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:03.524396 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:03.524403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:03.524717 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:03.525123 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.525136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.525924 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.525928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.526788 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.526797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.527295 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.527299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.527456 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.527963 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:03.527978 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:03.529351 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:03.531118 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:03.531122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:03.531132 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:03.531357 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:03.531648 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:03.531650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:03.531651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:03.531763 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:03.532205 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:03.532212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:03.532460 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 14:26:03.532691 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 14:26:03.532957 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":93,"threads_started":1}
I20260620 14:26:03.534734 4807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37995
I20260620 14:26:03.534955 4807 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:03.535346 4807 heartbeater.cc:511] Master 127.0.98.62:37995 requested a full tablet report, sending...
I20260620 14:26:03.536269 4484 ts_manager.cc:194] Registered new tserver with Master: 29e77d40b84d4c35b19fd00644cf33fb (127.0.98.2:34153)
I20260620 14:26:03.536798 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47769
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:26:03.553134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:03.654284 4812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:03.654517 4812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:03.654580 4812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:03.658458 4812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:03.658527 4812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:03.658614 4812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:03.663177 4812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:03.664003 4812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:03.664871 4812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:03.673187 4820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:03.673219 4818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:03.673250 4817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:03.673393 4812 server_base.cc:1034] running on GCE node
I20260620 14:26:03.673921 4812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:03.674394 4812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:03.675546 4812 hybrid_clock.cc:648] HybridClock initialized: now 1781965563675533 us; error 30 us; skew 500 ppm
Jun 20 14:26:03 dist-test-slave-pk3j krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:03.678239 4812 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:26:03.678859 4812 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:03.678946 4812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:03.679147 4812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:03.680867 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "56aaea39f14d4dcc95bd0d1b09480d6d"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "b9314273ebab60f22556f23bb1175fbc"
server_key_iv: "ec02194673b236c61da3934b016fc585"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.681416 4812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "56aaea39f14d4dcc95bd0d1b09480d6d"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "b9314273ebab60f22556f23bb1175fbc"
server_key_iv: "ec02194673b236c61da3934b016fc585"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.684520 4812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:26:03.686789 4826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:03.688058 4812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:03.688180 4812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "56aaea39f14d4dcc95bd0d1b09480d6d"
format_stamp: "Formatted at 2026-06-20 14:26:03 on dist-test-slave-pk3j"
server_key: "b9314273ebab60f22556f23bb1175fbc"
server_key_iv: "ec02194673b236c61da3934b016fc585"
server_key_version: "encryptionkey@0"
I20260620 14:26:03.688292 4812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:03.719398 4812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:03.722538 4812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:03.722720 4812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:03.723249 4812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:03.724009 4812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:03.724082 4812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:03.724146 4812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:03.724195 4812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:03.732795 4812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32951
I20260620 14:26:03.732821 4939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32951 every 8 connection(s)
I20260620 14:26:03.733345 4812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:03.738271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 4812
I20260620 14:26:03.738401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:26:03.738654 392 external_mini_cluster.cc:1428] Setting key 931b6859c1814ad80f7cd8119b3d7596
Jun 20 14:26:03 dist-test-slave-pk3j krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:03.745836 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.734881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49781 (local address 127.0.98.62:37995)
0620 14:26:03.735014 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.735018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.735752 (+ 734us) server_negotiation.cc:407] Connection header received
0620 14:26:03.737058 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.737061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.737115 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.737194 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:03.738254 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.738896 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.740089 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.740293 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.740418 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.742966 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:03.742970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:03.742972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:03.742994 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:03.744190 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:03.744622 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:03.744625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:03.744626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:03.744692 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:03.744964 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:03.744966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:03.744967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:03.745120 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:03.745190 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.745652 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.745733 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:26:03.746556 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.735214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37995 (local address 127.0.98.3:49781)
0620 14:26:03.735614 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:03.735652 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:03.736858 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:03.737381 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:03.737387 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:03.737680 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:03.738117 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.738129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.739009 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.739014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.739970 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.739979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.740467 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.740471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.740642 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.741169 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:03.741184 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:03.742811 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:03.744279 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:03.744282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:03.744292 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:03.744542 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:03.744763 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:03.744765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:03.744767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:03.744891 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:03.745238 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:03.745244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:03.745455 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 14:26:03.745606 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:26:03.745800 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":109,"threads_started":1}
I20260620 14:26:03.747572 4940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37995
I20260620 14:26:03.747758 4940 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:03.748071 4940 heartbeater.cc:511] Master 127.0.98.62:37995 requested a full tablet report, sending...
I20260620 14:26:03.748871 4484 ts_manager.cc:194] Registered new tserver with Master: 56aaea39f14d4dcc95bd0d1b09480d6d (127.0.98.3:32951)
I20260620 14:26:03.749425 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49781
I20260620 14:26:03.752027 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:26:03.759781 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.753751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.0.98.62:37995)
0620 14:26:03.753853 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.753857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.754033 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:26:03.754113 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.754115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.754146 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.754212 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:03.754780 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.755218 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.755920 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.756121 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.756191 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.757370 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:03.757373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:03.757374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:03.757393 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:03.758556 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:03.758909 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:03.758911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:03.758912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:03.758977 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:03.759199 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:03.759201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:03.759202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:03.759367 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:03.759456 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.759620 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.759693 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:26:03.762365 4484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:26:03.765100 4484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:26:03.778136 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.772351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32951 (local address 127.0.0.1:60162)
0620 14:26:03.772940 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:03.772982 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:03.773118 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:03.773921 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:03.773926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:03.773948 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:03.774054 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.774059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.775396 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.775402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.776691 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.776701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.777666 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.777669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.777765 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.777796 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:26:03.777900 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:26:03.777999 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":54,"spinlock_wait_cycles":5248,"thread_start_us":76,"threads_started":1}
I20260620 14:26:03.778167 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.772865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.0.98.3:32951)
0620 14:26:03.773452 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.773463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.773503 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:26:03.773575 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.773580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.773763 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.773915 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:03.774171 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.775265 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.776819 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.777524 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.777638 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.777758 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.777940 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.778029 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":80,"threads_started":1}
I20260620 14:26:03.778453 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.773040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.0.98.2:34153)
0620 14:26:03.773575 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.773584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.773901 (+ 317us) server_negotiation.cc:407] Connection header received
0620 14:26:03.773972 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.773977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.774154 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.774314 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:03.774505 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.775308 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.776875 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.777560 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.777677 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.777798 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.778199 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.778305 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":85,"threads_started":1}
I20260620 14:26:03.778627 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.772614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34153 (local address 127.0.0.1:45622)
0620 14:26:03.773709 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:03.773723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:03.773806 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:03.774288 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:03.774291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:03.774307 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:03.774397 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.774403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.775441 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.775444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.776738 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.776747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.777867 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.777872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.777954 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.777973 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:26:03.778035 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:26:03.778087 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":28,"thread_start_us":95,"threads_started":1}
I20260620 14:26:03.778678 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.772201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35575 (local address 127.0.0.1:40778)
0620 14:26:03.772932 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:03.772979 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:03.773120 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:03.774588 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:03.774590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:03.774607 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:03.774678 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.774683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.775943 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.775946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.777034 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.777043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.778217 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.778222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.778381 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.778397 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:26:03.778489 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:26:03.778556 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"spinlock_wait_cycles":11136,"thread_start_us":72,"threads_started":1}
I20260620 14:26:03.778754 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.772352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.0.98.1:35575)
0620 14:26:03.774195 (+ 1843us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.774207 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.774228 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:26:03.774294 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.774299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.774462 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.774616 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:03.774755 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.775797 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.777155 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.777930 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.778026 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.778151 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.778519 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.778611 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1728,"thread_start_us":107,"threads_started":1}
I20260620 14:26:03.781144 4608 tablet_service.cc:1467] Processing CreateTablet for tablet bc0b9460014944eab4bbb77664a5ed37 (DEFAULT_TABLE table=test-table [id=bc633069b16d427c9bba5c45b1cc7c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:03.781263 4874 tablet_service.cc:1467] Processing CreateTablet for tablet bc0b9460014944eab4bbb77664a5ed37 (DEFAULT_TABLE table=test-table [id=bc633069b16d427c9bba5c45b1cc7c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:03.781641 4741 tablet_service.cc:1467] Processing CreateTablet for tablet bc0b9460014944eab4bbb77664a5ed37 (DEFAULT_TABLE table=test-table [id=bc633069b16d427c9bba5c45b1cc7c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:03.782182 4608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0b9460014944eab4bbb77664a5ed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:03.782212 4874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0b9460014944eab4bbb77664a5ed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:03.782465 4741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc0b9460014944eab4bbb77664a5ed37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:03.787628 4957 tablet_bootstrap.cc:492] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c: Bootstrap starting.
I20260620 14:26:03.788859 4958 tablet_bootstrap.cc:492] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: Bootstrap starting.
I20260620 14:26:03.790238 4957 tablet_bootstrap.cc:654] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:03.790886 4958 tablet_bootstrap.cc:654] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:03.791539 4957 log.cc:824] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:03.792071 4958 log.cc:824] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:03.792202 4959 tablet_bootstrap.cc:492] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: Bootstrap starting.
I20260620 14:26:03.793534 4957 tablet_bootstrap.cc:492] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c: No bootstrap required, opened a new log
I20260620 14:26:03.793743 4957 ts_tablet_manager.cc:1397] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:26:03.794025 4958 tablet_bootstrap.cc:492] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: No bootstrap required, opened a new log
I20260620 14:26:03.794024 4959 tablet_bootstrap.cc:654] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:03.794241 4958 ts_tablet_manager.cc:1397] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:26:03.795152 4959 log.cc:824] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:03.796844 4959 tablet_bootstrap.cc:492] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: No bootstrap required, opened a new log
I20260620 14:26:03.797017 4959 ts_tablet_manager.cc:1397] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:26:03.797416 4957 raft_consensus.cc:348] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } } peers { permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 } }
I20260620 14:26:03.797649 4957 raft_consensus.cc:374] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:03.797700 4957 raft_consensus.cc:729] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad991c0fcfe490d8cd6a69fd268621c, State: Initialized, Role: FOLLOWER
I20260620 14:26:03.798130 4957 consensus_queue.cc:260] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } } peers { permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 } }
I20260620 14:26:03.798079 4958 raft_consensus.cc:348] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } } peers { permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 } }
I20260620 14:26:03.798313 4958 raft_consensus.cc:374] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:03.798372 4958 raft_consensus.cc:729] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e77d40b84d4c35b19fd00644cf33fb, State: Initialized, Role: FOLLOWER
I20260620 14:26:03.798743 4958 consensus_queue.cc:260] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } } peers { permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 } }
I20260620 14:26:03.798998 4957 ts_tablet_manager.cc:1428] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:26:03.799405 4674 heartbeater.cc:503] Master 127.0.98.62:37995 was elected leader, sending a full tablet report...
I20260620 14:26:03.799408 4807 heartbeater.cc:503] Master 127.0.98.62:37995 was elected leader, sending a full tablet report...
I20260620 14:26:03.800695 4958 ts_tablet_manager.cc:1428] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:26:03.801359 4959 raft_consensus.cc:348] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } } peers { permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 } }
I20260620 14:26:03.801559 4959 raft_consensus.cc:374] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:03.801614 4959 raft_consensus.cc:729] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56aaea39f14d4dcc95bd0d1b09480d6d, State: Initialized, Role: FOLLOWER
I20260620 14:26:03.801934 4959 consensus_queue.cc:260] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } } peers { permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 } }
I20260620 14:26:03.802785 4959 ts_tablet_manager.cc:1428] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:26:03.803273 4940 heartbeater.cc:503] Master 127.0.98.62:37995 was elected leader, sending a full tablet report...
I20260620 14:26:03.823370 4963 raft_consensus.cc:482] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:03.823510 4963 raft_consensus.cc:504] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } } peers { permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 } }
I20260620 14:26:03.824376 4963 leader_election.cc:288] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56aaea39f14d4dcc95bd0d1b09480d6d (127.0.98.3:32951), 29e77d40b84d4c35b19fd00644cf33fb (127.0.98.2:34153)
I20260620 14:26:03.829059 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.824640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47309 (local address 127.0.98.2:34153)
0620 14:26:03.824742 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.824747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.824986 (+ 239us) server_negotiation.cc:407] Connection header received
0620 14:26:03.825030 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.825035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.825086 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.825149 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:03.825428 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.826190 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.827857 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.828480 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.828586 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.828640 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.828885 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.828947 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:26:03.829059 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.824629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53949 (local address 127.0.98.3:32951)
0620 14:26:03.824751 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.824757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.824800 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:26:03.824935 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.824938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.825005 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.825086 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:03.825428 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.826184 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.827853 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.828484 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.828586 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.828640 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.828885 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.828947 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:26:03.829090 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.824568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32951 (local address 127.0.98.1:53949)
0620 14:26:03.824729 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:03.824745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:03.824827 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:03.825106 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:03.825108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:03.825120 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:03.825314 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.825321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.826293 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.826296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.827726 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.827737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.828598 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.828602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.828734 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.828774 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:26:03.828851 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:26:03.828946 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":149}
I20260620 14:26:03.829099 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.824572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34153 (local address 127.0.98.1:47309)
0620 14:26:03.824873 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:03.824888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:03.824966 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:03.825170 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:03.825172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:03.825183 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:03.825327 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.825332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.826293 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.826296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.827726 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:03.827737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.828598 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.828602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:03.828731 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.828782 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:26:03.828851 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:26:03.828946 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":183,"thread_start_us":63,"threads_started":1}
W20260620 14:26:03.829319 4675 tablet.cc:2369] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:26:03.829638 4761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0b9460014944eab4bbb77664a5ed37" candidate_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29e77d40b84d4c35b19fd00644cf33fb" is_pre_election: true
I20260620 14:26:03.829887 4761 raft_consensus.cc:2454] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad991c0fcfe490d8cd6a69fd268621c in term 0.
I20260620 14:26:03.830238 4894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0b9460014944eab4bbb77664a5ed37" candidate_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" is_pre_election: true
I20260620 14:26:03.830317 4561 leader_election.cc:302] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29e77d40b84d4c35b19fd00644cf33fb, 4ad991c0fcfe490d8cd6a69fd268621c; no voters:
I20260620 14:26:03.830493 4894 raft_consensus.cc:2454] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad991c0fcfe490d8cd6a69fd268621c in term 0.
I20260620 14:26:03.830567 4963 raft_consensus.cc:2781] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:03.830636 4963 raft_consensus.cc:482] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:03.830687 4963 raft_consensus.cc:3037] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:03.831655 4963 raft_consensus.cc:504] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } } peers { permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 } }
I20260620 14:26:03.831921 4963 leader_election.cc:288] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [CANDIDATE]: Term 1 election: Requested vote from peers 56aaea39f14d4dcc95bd0d1b09480d6d (127.0.98.3:32951), 29e77d40b84d4c35b19fd00644cf33fb (127.0.98.2:34153)
I20260620 14:26:03.832278 4761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0b9460014944eab4bbb77664a5ed37" candidate_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29e77d40b84d4c35b19fd00644cf33fb"
I20260620 14:26:03.832301 4894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0b9460014944eab4bbb77664a5ed37" candidate_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d"
I20260620 14:26:03.832365 4761 raft_consensus.cc:3037] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:03.832392 4894 raft_consensus.cc:3037] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:03.833299 4894 raft_consensus.cc:2454] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad991c0fcfe490d8cd6a69fd268621c in term 1.
I20260620 14:26:03.833323 4761 raft_consensus.cc:2454] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad991c0fcfe490d8cd6a69fd268621c in term 1.
I20260620 14:26:03.833650 4563 leader_election.cc:302] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad991c0fcfe490d8cd6a69fd268621c, 56aaea39f14d4dcc95bd0d1b09480d6d; no voters:
I20260620 14:26:03.833854 4963 raft_consensus.cc:2781] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:03.834080 4963 raft_consensus.cc:686] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 1 LEADER]: Becoming Leader. State: Replica: 4ad991c0fcfe490d8cd6a69fd268621c, State: Running, Role: LEADER
I20260620 14:26:03.834374 4963 consensus_queue.cc:237] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } } peers { permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } } peers { permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 } }
I20260620 14:26:03.837036 4483 catalog_manager.cc:5495] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ad991c0fcfe490d8cd6a69fd268621c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad991c0fcfe490d8cd6a69fd268621c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 } health_report { overall_health: UNKNOWN } } }
I20260620 14:26:03.856282 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.853245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.0.98.1:35575)
0620 14:26:03.853366 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:03.853370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:03.853403 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:26:03.853558 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:03.853560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:03.853603 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:03.853684 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:26:03.853878 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.854316 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.855103 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:03.855294 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:03.855366 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:03.855595 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:26:03.855921 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:26:03.856009 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:03.856067 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:26:03.856107 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:26:03.861115 4894 raft_consensus.cc:1261] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d [term 1 FOLLOWER]: Refusing update from remote peer 4ad991c0fcfe490d8cd6a69fd268621c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:03.861119 4761 raft_consensus.cc:1261] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb [term 1 FOLLOWER]: Refusing update from remote peer 4ad991c0fcfe490d8cd6a69fd268621c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:03.861778 4963 consensus_queue.cc:1035] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [LEADER]: Connected to new peer: Peer: permanent_uuid: "56aaea39f14d4dcc95bd0d1b09480d6d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:03.861954 4967 consensus_queue.cc:1035] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [LEADER]: Connected to new peer: Peer: permanent_uuid: "29e77d40b84d4c35b19fd00644cf33fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:03.869678 4971 mvcc.cc:205] Tried to move back new op lower bound from 7298930949569449984 to 7298930949467799552. Current Snapshot: MvccSnapshot[applied={T|T < 7298930949569449984}]
I20260620 14:26:03.875276 4483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:03.875424 4483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:03.877458 4483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a48dea6732d04eaba9539956a72b834e: Sending DeleteTablet for 3 replicas of tablet bc0b9460014944eab4bbb77664a5ed37
I20260620 14:26:03.878140 4608 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0b9460014944eab4bbb77664a5ed37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40778
I20260620 14:26:03.878165 4874 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0b9460014944eab4bbb77664a5ed37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60162
I20260620 14:26:03.878382 4741 tablet_service.cc:1514] Processing DeleteTablet for tablet bc0b9460014944eab4bbb77664a5ed37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45622
I20260620 14:26:03.878758 4980 tablet_replica.cc:330] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c: stopping tablet replica
I20260620 14:26:03.878948 4982 tablet_replica.cc:330] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: stopping tablet replica
I20260620 14:26:03.879055 4980 raft_consensus.cc:2229] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:26:03.879261 4982 raft_consensus.cc:2229] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:03.879285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 4546
I20260620 14:26:03.879422 4980 raft_consensus.cc:2258] T bc0b9460014944eab4bbb77664a5ed37 P 4ad991c0fcfe490d8cd6a69fd268621c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:03.879575 4982 raft_consensus.cc:2258] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:03.878827 4981 tablet_replica.cc:330] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: stopping tablet replica
I20260620 14:26:03.881076 4982 ts_tablet_manager.cc:1905] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:03.881224 4981 raft_consensus.cc:2229] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:03.881462 4981 raft_consensus.cc:2258] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:03.883204 4981 ts_tablet_manager.cc:1905] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:03.884230 4982 ts_tablet_manager.cc:1918] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:03.884327 4982 log.cc:1196] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/bc0b9460014944eab4bbb77664a5ed37
I20260620 14:26:03.884689 4982 ts_tablet_manager.cc:1939] T bc0b9460014944eab4bbb77664a5ed37 P 29e77d40b84d4c35b19fd00644cf33fb: Deleting consensus metadata
I20260620 14:26:03.885147 4981 ts_tablet_manager.cc:1918] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:03.885265 4981 log.cc:1196] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/bc0b9460014944eab4bbb77664a5ed37
I20260620 14:26:03.885560 4981 ts_tablet_manager.cc:1939] T bc0b9460014944eab4bbb77664a5ed37 P 56aaea39f14d4dcc95bd0d1b09480d6d: Deleting consensus metadata
I20260620 14:26:03.885823 4468 catalog_manager.cc:4841] TS 29e77d40b84d4c35b19fd00644cf33fb (127.0.98.2:34153): tablet bc0b9460014944eab4bbb77664a5ed37 (table test-table [id=bc633069b16d427c9bba5c45b1cc7c58]) successfully deleted
W20260620 14:26:03.886474 4469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35575 (error 108)
I20260620 14:26:03.886585 4470 catalog_manager.cc:4841] TS 56aaea39f14d4dcc95bd0d1b09480d6d (127.0.98.3:32951): tablet bc0b9460014944eab4bbb77664a5ed37 (table test-table [id=bc633069b16d427c9bba5c45b1cc7c58]) successfully deleted
I20260620 14:26:03.887342 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:03.887098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35575 (local address 127.0.0.1:40788)
0620 14:26:03.887230 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:03.887283 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 14:26:03.887478 4469 catalog_manager.cc:4584] TS 4ad991c0fcfe490d8cd6a69fd268621c (127.0.98.1:35575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc0b9460014944eab4bbb77664a5ed37: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35575: connect: Connection refused (error 111)
I20260620 14:26:03.887532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 4679
I20260620 14:26:03.893254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 4812
I20260620 14:26:03.901984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 4453
2026-06-20T14:26:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2094 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:26:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:03Z Disabled control of system clock
I20260620 14:26:03.936151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35491
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:26:04.042529 4987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:04.042737 4987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:04.042769 4987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:04.046561 4987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:04.046607 4987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:04.046622 4987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:04.046667 4987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:04.046700 4987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:04.051096 4987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:04.051959 4987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:04.053189 4987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:26:04.060612 4987 server_base.cc:1034] running on GCE node
W20260620 14:26:04.060575 4993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:04.060632 4992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:04.060575 4995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:04.061342 4987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:04.062294 4987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:04.063477 4987 hybrid_clock.cc:648] HybridClock initialized: now 1781965564063463 us; error 37 us; skew 500 ppm
I20260620 14:26:04.065065 4987 webserver.cc:460] Webserver started at http://127.0.98.62:38381/ using document root <none> and password file <none>
I20260620 14:26:04.065613 4987 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:04.065701 4987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:04.065888 4987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:04.067727 4987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "3fa3893003614b5aae1491c5437ef100"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "69f5980cda41138e76c269df1390a87b"
server_key_iv: "ae74deb2bbf5e140902270470c983445"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.068248 4987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "3fa3893003614b5aae1491c5437ef100"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "69f5980cda41138e76c269df1390a87b"
server_key_iv: "ae74deb2bbf5e140902270470c983445"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.071463 4987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:04.073632 5001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.074692 4987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:04.074822 4987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "3fa3893003614b5aae1491c5437ef100"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "69f5980cda41138e76c269df1390a87b"
server_key_iv: "ae74deb2bbf5e140902270470c983445"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.074924 4987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:04.079084 4987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:04.079600 4987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:04.079770 4987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:04.085846 4987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35491
I20260620 14:26:04.085875 5053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35491 every 8 connection(s)
I20260620 14:26:04.086640 4987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:04.089437 5054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:04.091154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 4987
I20260620 14:26:04.091274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:04.091504 392 external_mini_cluster.cc:1428] Setting key 43dfb226f06b39a45ce843f539ba8251
I20260620 14:26:04.095088 5054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100: Bootstrap starting.
I20260620 14:26:04.096921 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.092741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.0.98.62:35491)
0620 14:26:04.093369 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.093388 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.093427 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:26:04.094167 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.094189 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.094590 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.094912 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.095596 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.095601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.095614 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.095625 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.095684 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.095704 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.095826 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.095885 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.096126 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
I20260620 14:26:04.097570 5054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:04.098272 5054 log.cc:824] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:04.099920 5054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100: No bootstrap required, opened a new log
I20260620 14:26:04.103219 5054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa3893003614b5aae1491c5437ef100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260620 14:26:04.103402 5054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:04.103474 5054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa3893003614b5aae1491c5437ef100, State: Initialized, Role: FOLLOWER
I20260620 14:26:04.103890 5054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa3893003614b5aae1491c5437ef100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260620 14:26:04.104027 5054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:04.104090 5054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:04.104187 5054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:04.104926 5054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa3893003614b5aae1491c5437ef100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260620 14:26:04.105278 5054 leader_election.cc:302] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fa3893003614b5aae1491c5437ef100; no voters:
I20260620 14:26:04.105553 5054 leader_election.cc:288] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:04.105684 5059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:04.105892 5059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [term 1 LEADER]: Becoming Leader. State: Replica: 3fa3893003614b5aae1491c5437ef100, State: Running, Role: LEADER
I20260620 14:26:04.106226 5059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa3893003614b5aae1491c5437ef100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260620 14:26:04.106678 5054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:04.107815 5060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fa3893003614b5aae1491c5437ef100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa3893003614b5aae1491c5437ef100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } } }
I20260620 14:26:04.107936 5060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:04.108322 5068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:04.108568 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fa3893003614b5aae1491c5437ef100. Latest consensus state: current_term: 1 leader_uuid: "3fa3893003614b5aae1491c5437ef100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa3893003614b5aae1491c5437ef100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } } }
I20260620 14:26:04.108708 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:04.110786 5068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:04.115727 5068 catalog_manager.cc:1347] Generated new cluster ID: c536d40e22ec45448b31f17c4e3363b1
I20260620 14:26:04.115785 5068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:04.138914 5068 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:04.139204 5068 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:04.150065 5068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100: Generated new TSK 0
I20260620 14:26:04.150725 5068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:26:04.166091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--builtin_ntp_servers=127.0.98.20:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:26:04.273458 5078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:04.273670 5078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:04.273738 5078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:04.277616 5078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:04.277657 5078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:04.277751 5078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:04.282413 5078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:04.283233 5078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:04.284389 5078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:04.293330 5084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:04.293330 5086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:04.293359 5083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:04.293401 5078 server_base.cc:1034] running on GCE node
I20260620 14:26:04.293866 5078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:04.294416 5078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:04.295593 5078 hybrid_clock.cc:648] HybridClock initialized: now 1781965564295582 us; error 28 us; skew 500 ppm
I20260620 14:26:04.297392 5078 webserver.cc:460] Webserver started at http://127.0.98.1:37079/ using document root <none> and password file <none>
I20260620 14:26:04.297901 5078 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:04.297988 5078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:04.298202 5078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:04.300019 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "346538b140cd4452a64528324d79cb67"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "97afbb8147a4d3e10d18582c558006cb"
server_key_iv: "abf00e42a3ac003c09f828f5f1af7c00"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.300546 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "346538b140cd4452a64528324d79cb67"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "97afbb8147a4d3e10d18582c558006cb"
server_key_iv: "abf00e42a3ac003c09f828f5f1af7c00"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.303867 5078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:26:04.306543 5092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.307735 5078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:04.307864 5078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "346538b140cd4452a64528324d79cb67"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "97afbb8147a4d3e10d18582c558006cb"
server_key_iv: "abf00e42a3ac003c09f828f5f1af7c00"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.307967 5078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:04.312403 5078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:04.312922 5078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:04.313098 5078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:04.313621 5078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:04.314429 5078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:04.314498 5078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.314563 5078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:04.314596 5078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.322786 5078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46541
I20260620 14:26:04.322840 5205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46541 every 8 connection(s)
I20260620 14:26:04.323621 5078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:26:04.324275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 5078
I20260620 14:26:04.324404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:26:04.324645 392 external_mini_cluster.cc:1428] Setting key bd8591ab6d8ef9cb273272067faa2ce1
I20260620 14:26:04.326654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--builtin_ntp_servers=127.0.98.20:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:26:04.330083 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.325312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54641 (local address 127.0.98.62:35491)
0620 14:26:04.325438 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.325442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.326636 (+ 1194us) server_negotiation.cc:407] Connection header received
0620 14:26:04.328396 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.328399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.328436 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.328509 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.329629 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.329633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.329635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.329647 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.329668 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.329677 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.329776 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.329914 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.329955 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:26:04.330904 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.325713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:54641)
0620 14:26:04.326478 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:04.326531 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:04.328208 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:04.328655 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:04.328662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:04.328741 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:04.329398 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:04.329399 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:04.329433 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:04.329494 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:04.329765 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:04.329774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:04.329780 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:26:04.329952 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:26:04.330164 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":144,"threads_started":1}
I20260620 14:26:04.332242 5206 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35491
I20260620 14:26:04.332453 5206 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:04.332804 5206 heartbeater.cc:511] Master 127.0.98.62:35491 requested a full tablet report, sending...
I20260620 14:26:04.334188 5018 ts_manager.cc:194] Registered new tserver with Master: 346538b140cd4452a64528324d79cb67 (127.0.98.1:46541)
W20260620 14:26:04.429323 5209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:04.429538 5209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:04.429603 5209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:04.433538 5209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:04.433579 5209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:04.433668 5209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:04.438126 5209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:04.438997 5209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:04.440207 5209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:04.448853 5214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:04.448894 5217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:04.448894 5215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:04.449076 5209 server_base.cc:1034] running on GCE node
I20260620 14:26:04.449508 5209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:04.449992 5209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:04.451148 5209 hybrid_clock.cc:648] HybridClock initialized: now 1781965564451129 us; error 32 us; skew 500 ppm
I20260620 14:26:04.452831 5209 webserver.cc:460] Webserver started at http://127.0.98.2:43523/ using document root <none> and password file <none>
I20260620 14:26:04.453401 5209 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:04.453460 5209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:04.453680 5209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:04.455500 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "f602ad7adc184f07a0a23e20060f38f2"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "3e5c3003cb70dbeb0db0b6406e6d0308"
server_key_iv: "6bfd55c08cd74ed06877bd95796a4f95"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.456068 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f602ad7adc184f07a0a23e20060f38f2"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "3e5c3003cb70dbeb0db0b6406e6d0308"
server_key_iv: "6bfd55c08cd74ed06877bd95796a4f95"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.459424 5209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:04.461577 5223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.462623 5209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:26:04.462747 5209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "f602ad7adc184f07a0a23e20060f38f2"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "3e5c3003cb70dbeb0db0b6406e6d0308"
server_key_iv: "6bfd55c08cd74ed06877bd95796a4f95"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.462848 5209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:04.466951 5209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:04.467478 5209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:04.467650 5209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:04.468127 5209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:04.468947 5209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:04.469015 5209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.469089 5209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:04.469125 5209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.477288 5209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34013
I20260620 14:26:04.477317 5336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34013 every 8 connection(s)
I20260620 14:26:04.478104 5209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:04.482211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 5209
I20260620 14:26:04.482321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:26:04.482575 392 external_mini_cluster.cc:1428] Setting key 14761a29e15af1c1279a9c6a44472922
I20260620 14:26:04.483664 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.479694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41473 (local address 127.0.98.62:35491)
0620 14:26:04.479822 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.479825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.480740 (+ 915us) server_negotiation.cc:407] Connection header received
0620 14:26:04.481921 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.481924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.481963 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.482097 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.483210 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.483214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.483216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.483227 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.483247 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.483257 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.483350 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.483493 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.483541 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:26:04.484403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--builtin_ntp_servers=127.0.98.20:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:26:04.484546 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.480021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:41473)
0620 14:26:04.480612 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:04.480645 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:04.481743 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:04.482237 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:04.482243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:04.482378 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:04.482973 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:04.482974 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:04.483006 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:04.483070 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:04.483347 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:04.483355 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:04.483362 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:26:04.483528 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 14:26:04.483730 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260620 14:26:04.485517 5337 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35491
I20260620 14:26:04.485740 5337 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:04.486121 5337 heartbeater.cc:511] Master 127.0.98.62:35491 requested a full tablet report, sending...
I20260620 14:26:04.487032 5018 ts_manager.cc:194] Registered new tserver with Master: f602ad7adc184f07a0a23e20060f38f2 (127.0.98.2:34013)
W20260620 14:26:04.586470 5340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:04.586709 5340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:04.586771 5340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:04.590548 5340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:04.590607 5340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:04.590698 5340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:04.595062 5340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:04.595860 5340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:04.597105 5340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:04.605736 5348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:04.605762 5345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:04.605783 5346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:04.605944 5340 server_base.cc:1034] running on GCE node
I20260620 14:26:04.606352 5340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:04.606891 5340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:04.608060 5340 hybrid_clock.cc:648] HybridClock initialized: now 1781965564608056 us; error 62 us; skew 500 ppm
I20260620 14:26:04.612411 5340 webserver.cc:460] Webserver started at http://127.0.98.3:36511/ using document root <none> and password file <none>
I20260620 14:26:04.612905 5340 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:04.612959 5340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:04.613185 5340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:04.615015 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "88a7c048ff7140c4a261e58e1bd9b73c"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "923cdf831f339070f3bbefeacd022c5b"
server_key_iv: "5570925552aa42d40f29a37e88e73ce7"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.615540 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "88a7c048ff7140c4a261e58e1bd9b73c"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "923cdf831f339070f3bbefeacd022c5b"
server_key_iv: "5570925552aa42d40f29a37e88e73ce7"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.618837 5340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:04.620837 5354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.621960 5340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:04.622092 5340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "88a7c048ff7140c4a261e58e1bd9b73c"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "923cdf831f339070f3bbefeacd022c5b"
server_key_iv: "5570925552aa42d40f29a37e88e73ce7"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.622188 5340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:04.626463 5340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:04.626973 5340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:04.627143 5340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:04.627626 5340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:04.628405 5340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:04.628470 5340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.628531 5340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:04.628577 5340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.636272 5340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44259
I20260620 14:26:04.636296 5467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44259 every 8 connection(s)
I20260620 14:26:04.636989 5340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:04.640054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 5340
I20260620 14:26:04.640173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:26:04.640470 392 external_mini_cluster.cc:1428] Setting key b816f5a93519ba5ad991c5c0e7280671
I20260620 14:26:04.642674 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.638565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51581 (local address 127.0.98.62:35491)
0620 14:26:04.638685 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.638688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.639576 (+ 888us) server_negotiation.cc:407] Connection header received
0620 14:26:04.641055 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.641058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.641098 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.641202 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.642265 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.642269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.642271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.642282 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.642299 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.642308 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.642408 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.642554 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.642587 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:26:04.643441 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.638872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:51581)
0620 14:26:04.639442 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:04.639476 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:04.640855 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:04.641387 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:04.641394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:04.641490 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:04.642040 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:04.642041 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:04.642075 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:04.642132 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:04.642427 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:04.642434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:04.642439 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:26:04.642565 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:26:04.642747 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
I20260620 14:26:04.644330 5468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35491
I20260620 14:26:04.644524 5468 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:04.644850 5468 heartbeater.cc:511] Master 127.0.98.62:35491 requested a full tablet report, sending...
I20260620 14:26:04.645747 5018 ts_manager.cc:194] Registered new tserver with Master: 88a7c048ff7140c4a261e58e1bd9b73c (127.0.98.3:44259)
I20260620 14:26:04.653802 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:26:04.656826 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.655506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.0.98.62:35491)
0620 14:26:04.655629 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.655633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.655807 (+ 174us) server_negotiation.cc:407] Connection header received
0620 14:26:04.655892 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.655894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.655918 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.655977 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.656531 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.656534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.656536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.656545 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.656559 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.656566 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.656629 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.656669 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.656694 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:26:04.659086 5018 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:26:04.661839 5018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:26:04.671751 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.668977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46541 (local address 127.0.0.1:54490)
0620 14:26:04.669260 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:04.669296 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:04.669376 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:04.670510 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:04.670515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:04.670576 (+ 61us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:04.671021 (+ 445us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:04.671025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:04.671042 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:04.671098 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:04.671450 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:04.671461 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:04.671469 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:26:04.671583 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:26:04.671639 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260620 14:26:04.671758 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.669106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.0.98.1:46541)
0620 14:26:04.669465 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.669477 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.669500 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:26:04.669553 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.669559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.669738 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.669873 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.671231 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.671236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.671246 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.671258 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.671320 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.671356 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.671460 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.671589 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.671641 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":189,"threads_started":1}
I20260620 14:26:04.672593 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.669600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44259 (local address 127.0.0.1:58624)
0620 14:26:04.670343 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:04.670353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:04.670416 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:04.671002 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:04.671005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:04.671026 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:04.671935 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:04.671937 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:04.671946 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:04.671973 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:04.672370 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:04.672373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:04.672376 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:26:04.672449 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:26:04.672481 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":76,"threads_started":1}
I20260620 14:26:04.672964 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.669907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.0.98.3:44259)
0620 14:26:04.670266 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.670275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.670430 (+ 155us) server_negotiation.cc:407] Connection header received
0620 14:26:04.670477 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.670482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.670620 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.670751 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.672115 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.672120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.672137 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.672147 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.672177 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.672196 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.672326 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.672743 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.672804 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":116,"threads_started":1}
I20260620 14:26:04.673549 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.669656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34013 (local address 127.0.0.1:37836)
0620 14:26:04.670677 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:04.670686 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:04.670778 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:04.671739 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:04.671741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:04.671753 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:04.672921 (+ 1168us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:04.672923 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:04.672932 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:04.672956 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:04.673327 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:04.673330 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:04.673333 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:26:04.673414 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:26:04.673446 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":88,"threads_started":1}
I20260620 14:26:04.673655 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.670006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.0.98.2:34013)
0620 14:26:04.670589 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.670595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.670856 (+ 261us) server_negotiation.cc:407] Connection header received
0620 14:26:04.670908 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.670913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.671065 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.671191 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.673105 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.673111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.673120 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.673130 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.673172 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.673191 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.673330 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.673437 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.673501 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":117,"threads_started":1}
I20260620 14:26:04.674656 5140 tablet_service.cc:1467] Processing CreateTablet for tablet c0c78faf3d2c4c13bcc04dacde667ebb (DEFAULT_TABLE table=test-table [id=9fe76d4502814beb9a130b61af364762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:04.675650 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c78faf3d2c4c13bcc04dacde667ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:04.675630 5402 tablet_service.cc:1467] Processing CreateTablet for tablet c0c78faf3d2c4c13bcc04dacde667ebb (DEFAULT_TABLE table=test-table [id=9fe76d4502814beb9a130b61af364762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:04.675789 5271 tablet_service.cc:1467] Processing CreateTablet for tablet c0c78faf3d2c4c13bcc04dacde667ebb (DEFAULT_TABLE table=test-table [id=9fe76d4502814beb9a130b61af364762]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:04.676499 5402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c78faf3d2c4c13bcc04dacde667ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:04.676676 5271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c78faf3d2c4c13bcc04dacde667ebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:04.682216 5485 tablet_bootstrap.cc:492] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2: Bootstrap starting.
I20260620 14:26:04.682972 5486 tablet_bootstrap.cc:492] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67: Bootstrap starting.
I20260620 14:26:04.682972 5487 tablet_bootstrap.cc:492] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: Bootstrap starting.
I20260620 14:26:04.684778 5485 tablet_bootstrap.cc:654] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:04.684928 5486 tablet_bootstrap.cc:654] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:04.684928 5487 tablet_bootstrap.cc:654] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:04.685570 5485 log.cc:824] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:04.685719 5486 log.cc:824] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:04.685719 5487 log.cc:824] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:04.687475 5487 tablet_bootstrap.cc:492] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: No bootstrap required, opened a new log
I20260620 14:26:04.687475 5485 tablet_bootstrap.cc:492] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2: No bootstrap required, opened a new log
I20260620 14:26:04.687516 5486 tablet_bootstrap.cc:492] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67: No bootstrap required, opened a new log
I20260620 14:26:04.687645 5485 ts_tablet_manager.cc:1397] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 14:26:04.687645 5487 ts_tablet_manager.cc:1397] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 14:26:04.687716 5486 ts_tablet_manager.cc:1397] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 14:26:04.690963 5486 raft_consensus.cc:348] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "f602ad7adc184f07a0a23e20060f38f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34013 } } peers { permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 } }
I20260620 14:26:04.691180 5486 raft_consensus.cc:374] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:04.691236 5486 raft_consensus.cc:729] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346538b140cd4452a64528324d79cb67, State: Initialized, Role: FOLLOWER
I20260620 14:26:04.691630 5486 consensus_queue.cc:260] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "f602ad7adc184f07a0a23e20060f38f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34013 } } peers { permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 } }
I20260620 14:26:04.691819 5487 raft_consensus.cc:348] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "f602ad7adc184f07a0a23e20060f38f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34013 } } peers { permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 } }
I20260620 14:26:04.692052 5487 raft_consensus.cc:374] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:04.692102 5487 raft_consensus.cc:729] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a7c048ff7140c4a261e58e1bd9b73c, State: Initialized, Role: FOLLOWER
I20260620 14:26:04.691973 5485 raft_consensus.cc:348] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "f602ad7adc184f07a0a23e20060f38f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34013 } } peers { permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 } }
I20260620 14:26:04.692168 5485 raft_consensus.cc:374] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:04.692219 5485 raft_consensus.cc:729] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f602ad7adc184f07a0a23e20060f38f2, State: Initialized, Role: FOLLOWER
I20260620 14:26:04.692390 5206 heartbeater.cc:503] Master 127.0.98.62:35491 was elected leader, sending a full tablet report...
I20260620 14:26:04.692499 5487 consensus_queue.cc:260] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "f602ad7adc184f07a0a23e20060f38f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34013 } } peers { permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 } }
I20260620 14:26:04.692569 5486 ts_tablet_manager.cc:1428] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 14:26:04.693060 5468 heartbeater.cc:503] Master 127.0.98.62:35491 was elected leader, sending a full tablet report...
I20260620 14:26:04.693346 5487 ts_tablet_manager.cc:1428] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 14:26:04.693610 5485 consensus_queue.cc:260] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "f602ad7adc184f07a0a23e20060f38f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34013 } } peers { permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 } }
I20260620 14:26:04.694270 5337 heartbeater.cc:503] Master 127.0.98.62:35491 was elected leader, sending a full tablet report...
I20260620 14:26:04.694703 5485 ts_tablet_manager.cc:1428] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
W20260620 14:26:04.728953 5338 tablet.cc:2369] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:26:04.742669 5493 raft_consensus.cc:482] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:04.742861 5493 raft_consensus.cc:504] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "f602ad7adc184f07a0a23e20060f38f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34013 } } peers { permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 } }
I20260620 14:26:04.743952 5493 leader_election.cc:288] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 346538b140cd4452a64528324d79cb67 (127.0.98.1:46541), 88a7c048ff7140c4a261e58e1bd9b73c (127.0.98.3:44259)
I20260620 14:26:04.745765 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.744252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55455 (local address 127.0.98.3:44259)
0620 14:26:04.744356 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.744361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.744593 (+ 232us) server_negotiation.cc:407] Connection header received
0620 14:26:04.744636 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.744639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.744673 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.744737 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.745374 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.745377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.745380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.745391 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.745411 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.745421 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.745508 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.745603 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.745640 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:26:04.745795 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.744188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44259 (local address 127.0.98.2:55455)
0620 14:26:04.744503 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:04.744514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:04.744582 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:04.744768 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:04.744770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:04.744781 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:04.745193 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:04.745194 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:04.745223 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:04.745272 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:04.745507 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:04.745510 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:04.745513 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:26:04.745604 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:26:04.745638 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":102,"threads_started":1}
I20260620 14:26:04.745973 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.744114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46541 (local address 127.0.98.2:33653)
0620 14:26:04.744277 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:04.744292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:04.744360 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:04.744811 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:04.744814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:04.744825 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:04.745438 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:04.745439 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:04.745447 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:04.745468 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:04.745722 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:04.745724 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:04.745729 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:26:04.745833 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:26:04.745872 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 14:26:04.746404 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.744188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33653 (local address 127.0.98.1:46541)
0620 14:26:04.744306 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.744313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.744332 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:26:04.744471 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.744476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.744516 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.744602 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.745586 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.745589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.745591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.745602 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.745620 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.745635 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.745722 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.746285 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.746319 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:26:04.746402 5422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c78faf3d2c4c13bcc04dacde667ebb" candidate_uuid: "f602ad7adc184f07a0a23e20060f38f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" is_pre_election: true
I20260620 14:26:04.746634 5422 raft_consensus.cc:2454] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f602ad7adc184f07a0a23e20060f38f2 in term 0.
I20260620 14:26:04.746930 5160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c78faf3d2c4c13bcc04dacde667ebb" candidate_uuid: "f602ad7adc184f07a0a23e20060f38f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346538b140cd4452a64528324d79cb67" is_pre_election: true
I20260620 14:26:04.747095 5226 leader_election.cc:302] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a7c048ff7140c4a261e58e1bd9b73c, f602ad7adc184f07a0a23e20060f38f2; no voters:
I20260620 14:26:04.747165 5160 raft_consensus.cc:2454] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f602ad7adc184f07a0a23e20060f38f2 in term 0.
I20260620 14:26:04.747350 5493 raft_consensus.cc:2781] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:04.747431 5493 raft_consensus.cc:482] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:04.747485 5493 raft_consensus.cc:3037] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:04.748339 5493 raft_consensus.cc:504] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "f602ad7adc184f07a0a23e20060f38f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34013 } } peers { permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 } }
I20260620 14:26:04.748627 5493 leader_election.cc:288] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [CANDIDATE]: Term 1 election: Requested vote from peers 346538b140cd4452a64528324d79cb67 (127.0.98.1:46541), 88a7c048ff7140c4a261e58e1bd9b73c (127.0.98.3:44259)
I20260620 14:26:04.748914 5422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c78faf3d2c4c13bcc04dacde667ebb" candidate_uuid: "f602ad7adc184f07a0a23e20060f38f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a7c048ff7140c4a261e58e1bd9b73c"
I20260620 14:26:04.749042 5422 raft_consensus.cc:3037] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:04.749328 5160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c78faf3d2c4c13bcc04dacde667ebb" candidate_uuid: "f602ad7adc184f07a0a23e20060f38f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "346538b140cd4452a64528324d79cb67"
I20260620 14:26:04.749445 5160 raft_consensus.cc:3037] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:04.750075 5422 raft_consensus.cc:2454] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f602ad7adc184f07a0a23e20060f38f2 in term 1.
I20260620 14:26:04.750339 5226 leader_election.cc:302] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a7c048ff7140c4a261e58e1bd9b73c, f602ad7adc184f07a0a23e20060f38f2; no voters:
I20260620 14:26:04.750471 5160 raft_consensus.cc:2454] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f602ad7adc184f07a0a23e20060f38f2 in term 1.
I20260620 14:26:04.750523 5493 raft_consensus.cc:2781] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:04.750754 5493 raft_consensus.cc:686] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 1 LEADER]: Becoming Leader. State: Replica: f602ad7adc184f07a0a23e20060f38f2, State: Running, Role: LEADER
I20260620 14:26:04.751039 5493 consensus_queue.cc:237] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } } peers { permanent_uuid: "f602ad7adc184f07a0a23e20060f38f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34013 } } peers { permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 } }
I20260620 14:26:04.753631 5018 catalog_manager.cc:5495] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f602ad7adc184f07a0a23e20060f38f2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f602ad7adc184f07a0a23e20060f38f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f602ad7adc184f07a0a23e20060f38f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 } health_report { overall_health: UNKNOWN } } }
I20260620 14:26:04.767416 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.765818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.0.98.2:34013)
0620 14:26:04.765919 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:04.765925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:04.765991 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:26:04.766213 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:04.766216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:04.766273 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:04.766360 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:04.767054 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:04.767057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:04.767060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:04.767071 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:04.767089 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:04.767098 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:04.767195 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:04.767275 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:26:04.767308 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:26:04.772142 5422 raft_consensus.cc:1261] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c [term 1 FOLLOWER]: Refusing update from remote peer f602ad7adc184f07a0a23e20060f38f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:04.772142 5160 raft_consensus.cc:1261] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67 [term 1 FOLLOWER]: Refusing update from remote peer f602ad7adc184f07a0a23e20060f38f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:04.772836 5495 consensus_queue.cc:1035] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a7c048ff7140c4a261e58e1bd9b73c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:04.773022 5493 consensus_queue.cc:1035] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "346538b140cd4452a64528324d79cb67" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:04.779539 5499 mvcc.cc:205] Tried to move back new op lower bound from 7298930953300733952 to 7298930953222438912. Current Snapshot: MvccSnapshot[applied={T|T < 7298930953300733952}]
I20260620 14:26:04.786644 5018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:04.786787 5018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:04.788825 5018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fa3893003614b5aae1491c5437ef100: Sending DeleteTablet for 3 replicas of tablet c0c78faf3d2c4c13bcc04dacde667ebb
I20260620 14:26:04.789464 5140 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c78faf3d2c4c13bcc04dacde667ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:04 UTC) from {username='slave'} at 127.0.0.1:54490
I20260620 14:26:04.789486 5402 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c78faf3d2c4c13bcc04dacde667ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:04 UTC) from {username='slave'} at 127.0.0.1:58624
I20260620 14:26:04.789515 5271 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c78faf3d2c4c13bcc04dacde667ebb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:04 UTC) from {username='slave'} at 127.0.0.1:37836
I20260620 14:26:04.789856 5509 tablet_replica.cc:330] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67: stopping tablet replica
I20260620 14:26:04.790141 5509 raft_consensus.cc:2229] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:04.790367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 5078
I20260620 14:26:04.790390 5509 raft_consensus.cc:2258] T c0c78faf3d2c4c13bcc04dacde667ebb P 346538b140cd4452a64528324d79cb67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:04.791023 5508 tablet_replica.cc:330] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: stopping tablet replica
I20260620 14:26:04.791263 5508 raft_consensus.cc:2229] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:04.791507 5508 raft_consensus.cc:2258] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:04.792883 5508 ts_tablet_manager.cc:1905] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:04.795094 5508 ts_tablet_manager.cc:1918] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:04.795195 5508 log.cc:1196] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/c0c78faf3d2c4c13bcc04dacde667ebb
I20260620 14:26:04.795503 5508 ts_tablet_manager.cc:1939] T c0c78faf3d2c4c13bcc04dacde667ebb P 88a7c048ff7140c4a261e58e1bd9b73c: Deleting consensus metadata
I20260620 14:26:04.795759 5510 tablet_replica.cc:330] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2: stopping tablet replica
I20260620 14:26:04.796042 5510 raft_consensus.cc:2229] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:26:04.796388 5510 raft_consensus.cc:2258] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:04.796648 5004 catalog_manager.cc:4841] TS 88a7c048ff7140c4a261e58e1bd9b73c (127.0.98.3:44259): tablet c0c78faf3d2c4c13bcc04dacde667ebb (table test-table [id=9fe76d4502814beb9a130b61af364762]) successfully deleted
I20260620 14:26:04.797655 5510 ts_tablet_manager.cc:1905] T c0c78faf3d2c4c13bcc04dacde667ebb P f602ad7adc184f07a0a23e20060f38f2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:26:04.798570 5002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46541 (error 108)
I20260620 14:26:04.798705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 5209
I20260620 14:26:04.800066 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.799761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46541 (local address 127.0.0.1:54492)
0620 14:26:04.799900 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:04.799975 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:26:04.800215 5002 catalog_manager.cc:4584] TS 346538b140cd4452a64528324d79cb67 (127.0.98.1:46541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0c78faf3d2c4c13bcc04dacde667ebb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46541: connect: Connection refused (error 111)
I20260620 14:26:04.804193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 5340
I20260620 14:26:04.805006 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:04.804321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34013 (local address 127.0.0.1:37860)
0620 14:26:04.804917 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:04.804956 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520}
I20260620 14:26:04.810232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 4987
2026-06-20T14:26:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (903 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:26:04.830615 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:26:04.832028 392 webserver.cc:460] Webserver started at https://127.0.0.1:40285/ using document root <none> and password file <none>
I20260620 14:26:04.832195 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:26:04.832448 392 webserver.cc:460] Webserver started at http://127.0.0.1:41063/ using document root <none> and password file <none>
2026-06-20T14:26:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:04Z Disabled control of system clock
I20260620 14:26:04.840356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33409
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33409
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40285/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/testchainca.cert with env {}
W20260620 14:26:04.938544 5517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:04.938791 5517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:04.938870 5517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:04.942638 5517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:04.942713 5517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:04.942744 5517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:04.942791 5517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:04.942826 5517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:26:04.942859 5517 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:26:04.942889 5517 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40285/jwks.json
I20260620 14:26:04.947671 5517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40285/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:04.948504 5517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:04.949362 5517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:04.956665 5525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:04.956691 5522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:04.956691 5523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:04.956946 5517 server_base.cc:1034] running on GCE node
I20260620 14:26:04.957490 5517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:04.958412 5517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:04.959604 5517 hybrid_clock.cc:648] HybridClock initialized: now 1781965564959588 us; error 29 us; skew 500 ppm
I20260620 14:26:04.961191 5517 webserver.cc:460] Webserver started at http://127.0.98.62:36571/ using document root <none> and password file <none>
I20260620 14:26:04.961745 5517 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:04.961829 5517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:04.962018 5517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:04.963721 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "e49209a7316c4b10907dfc6739250741"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "1f8addc1a2db7fd576193acad60195c1"
server_key_iv: "d3c3443abb0f1381c92aa1f0c62336e4"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.964226 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "e49209a7316c4b10907dfc6739250741"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "1f8addc1a2db7fd576193acad60195c1"
server_key_iv: "d3c3443abb0f1381c92aa1f0c62336e4"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.967465 5517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 14:26:04.969559 5531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:04.970582 5517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 14:26:04.970706 5517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "e49209a7316c4b10907dfc6739250741"
format_stamp: "Formatted at 2026-06-20 14:26:04 on dist-test-slave-pk3j"
server_key: "1f8addc1a2db7fd576193acad60195c1"
server_key_iv: "d3c3443abb0f1381c92aa1f0c62336e4"
server_key_version: "encryptionkey@0"
I20260620 14:26:04.970803 5517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:04.988519 5517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:04.989024 5517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:04.989193 5517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:04.995257 5517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33409
I20260620 14:26:04.995266 5583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33409 every 8 connection(s)
I20260620 14:26:04.996023 5517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:04.998863 5584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:05.003517 5584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741: Bootstrap starting.
I20260620 14:26:05.005119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 5517
I20260620 14:26:05.005249 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:05.005504 392 external_mini_cluster.cc:1428] Setting key 35a0f7eb88f155ff5c3310e0fc2bbfeb
I20260620 14:26:05.005872 5584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:05.006721 5584 log.cc:824] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:05.008437 5584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741: No bootstrap required, opened a new log
I20260620 14:26:05.011574 5584 raft_consensus.cc:348] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49209a7316c4b10907dfc6739250741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260620 14:26:05.011798 5584 raft_consensus.cc:374] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:05.011857 5584 raft_consensus.cc:729] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49209a7316c4b10907dfc6739250741, State: Initialized, Role: FOLLOWER
I20260620 14:26:05.012295 5584 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49209a7316c4b10907dfc6739250741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260620 14:26:05.012439 5584 raft_consensus.cc:388] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:05.012527 5584 raft_consensus.cc:482] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:05.012641 5584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:05.013583 5584 raft_consensus.cc:504] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49209a7316c4b10907dfc6739250741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260620 14:26:05.013687 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:05.006486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.0.98.62:33409)
0620 14:26:05.007116 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:05.007135 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:05.007173 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:26:05.007835 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:05.007852 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:05.008139 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:05.008446 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:05.008808 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.009749 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.010597 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.011031 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.011189 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.012072 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:05.012077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:05.012088 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:05.012452 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:05.012518 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:05.012545 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:05.012651 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:05.012774 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:26:05.013014 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260620 14:26:05.013947 5584 leader_election.cc:302] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e49209a7316c4b10907dfc6739250741; no voters:
I20260620 14:26:05.014233 5584 leader_election.cc:288] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:05.014328 5589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:05.014551 5589 raft_consensus.cc:686] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [term 1 LEADER]: Becoming Leader. State: Replica: e49209a7316c4b10907dfc6739250741, State: Running, Role: LEADER
I20260620 14:26:05.014856 5589 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49209a7316c4b10907dfc6739250741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } }
I20260620 14:26:05.015300 5584 sys_catalog.cc:564] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:05.015954 5590 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49209a7316c4b10907dfc6739250741. Latest consensus state: current_term: 1 leader_uuid: "e49209a7316c4b10907dfc6739250741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49209a7316c4b10907dfc6739250741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } } }
I20260620 14:26:05.016100 5590 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:05.016245 5591 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e49209a7316c4b10907dfc6739250741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49209a7316c4b10907dfc6739250741" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33409 } } }
I20260620 14:26:05.016501 5591 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:05.016662 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:26:05.018263 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:05.020452 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:05.025553 5594 catalog_manager.cc:1347] Generated new cluster ID: c6ae3f90296840c09889857e71f7a70b
I20260620 14:26:05.025661 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:05.041402 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:05.042459 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:05.053226 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e49209a7316c4b10907dfc6739250741: Generated new TSK 0
I20260620 14:26:05.053889 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:26:05.077662 5614 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:26:05.077764 5614 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:26:05.078231 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:05.062043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.0.98.62:33409)
0620 14:26:05.062174 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:05.062178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:05.062335 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:26:05.062513 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:05.062517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:05.062568 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:05.062673 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:26:05.062910 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.063684 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.064479 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.064812 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.064912 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.065237 (+ 325us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:26:05.077898 (+ 12661us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:26:05.078115 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38960: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 14:26:05.078299 5587 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38960: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:26:05.079180 5613 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:05.061948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33409 (local address 127.0.0.1:38960)
0620 14:26:05.062244 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:05.062258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:05.062378 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:05.062686 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:05.062688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:05.062694 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:26:05.062815 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:05.062819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.063789 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.063791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:05.064378 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:05.064384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.064929 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.064945 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:05.065016 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.065052 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:26:05.078058 (+ 13006us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:26:05.078206 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":222,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:26:05.079800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 5517
2026-06-20T14:26:05Z chronyd exiting
I20260620 14:26:05.116480 392 test_util.cc:175] -----------------------------------------------
I20260620 14:26:05.116559 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781965536974027-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (287 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:26:05.117605 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:26:05.118398 392 webserver.cc:460] Webserver started at https://127.0.0.1:36539/ using document root <none> and password file <none>
I20260620 14:26:05.118516 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:26:05.118757 392 webserver.cc:460] Webserver started at http://127.0.0.1:42447/ using document root <none> and password file <none>
2026-06-20T14:26:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:05Z Disabled control of system clock
I20260620 14:26:05.127038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41205
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41205
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36539/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/testchainca.cert with env {}
W20260620 14:26:05.232107 5620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:05.232496 5620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:05.232538 5620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:05.236406 5620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:05.236455 5620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:05.236471 5620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:05.236487 5620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:05.236502 5620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:26:05.236517 5620 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:26:05.236531 5620 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36539/jwks.json
I20260620 14:26:05.241556 5620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41205
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36539/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:05.242435 5620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:05.243314 5620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:05.251123 5626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:05.251261 5620 server_base.cc:1034] running on GCE node
W20260620 14:26:05.251138 5625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:05.251196 5628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:05.251907 5620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:05.252939 5620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:05.254120 5620 hybrid_clock.cc:648] HybridClock initialized: now 1781965565254089 us; error 43 us; skew 500 ppm
I20260620 14:26:05.255897 5620 webserver.cc:460] Webserver started at http://127.0.98.62:41401/ using document root <none> and password file <none>
I20260620 14:26:05.256440 5620 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:05.256538 5620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:05.256752 5620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:05.258736 5620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "596c80a5e34845b89bcdaa21f048bd1e"
format_stamp: "Formatted at 2026-06-20 14:26:05 on dist-test-slave-pk3j"
server_key: "37dc963a11e4576c9d01dff783daeefb"
server_key_iv: "1183d23d68b207232b163309114cdbe7"
server_key_version: "encryptionkey@0"
I20260620 14:26:05.259289 5620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "596c80a5e34845b89bcdaa21f048bd1e"
format_stamp: "Formatted at 2026-06-20 14:26:05 on dist-test-slave-pk3j"
server_key: "37dc963a11e4576c9d01dff783daeefb"
server_key_iv: "1183d23d68b207232b163309114cdbe7"
server_key_version: "encryptionkey@0"
I20260620 14:26:05.262682 5620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:05.265178 5634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:05.266335 5620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:26:05.266467 5620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "596c80a5e34845b89bcdaa21f048bd1e"
format_stamp: "Formatted at 2026-06-20 14:26:05 on dist-test-slave-pk3j"
server_key: "37dc963a11e4576c9d01dff783daeefb"
server_key_iv: "1183d23d68b207232b163309114cdbe7"
server_key_version: "encryptionkey@0"
I20260620 14:26:05.266597 5620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:05.294835 5620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:05.295450 5620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:05.295639 5620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:05.301931 5620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41205
I20260620 14:26:05.301959 5686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41205 every 8 connection(s)
I20260620 14:26:05.302882 5620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:05.305771 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:05.310874 5687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e: Bootstrap starting.
I20260620 14:26:05.312094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 5620
I20260620 14:26:05.312213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:05.312482 392 external_mini_cluster.cc:1428] Setting key 1df6bc103bce7d46b72bf5dda9f0c4d1
I20260620 14:26:05.313620 5687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:05.314342 5687 log.cc:824] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:05.316200 5687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e: No bootstrap required, opened a new log
I20260620 14:26:05.319648 5687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596c80a5e34845b89bcdaa21f048bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41205 } }
I20260620 14:26:05.319852 5687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:05.319921 5687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596c80a5e34845b89bcdaa21f048bd1e, State: Initialized, Role: FOLLOWER
I20260620 14:26:05.320401 5687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596c80a5e34845b89bcdaa21f048bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41205 } }
I20260620 14:26:05.320565 5687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:05.320634 5687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:05.320751 5687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:05.321482 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:05.313542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.0.98.62:41205)
0620 14:26:05.314138 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:05.314152 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:05.314185 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:26:05.314925 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:05.314947 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:05.315314 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:05.315631 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:05.316086 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.317046 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.317975 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.318518 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.318682 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.319534 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:05.319543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:05.319561 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:05.320028 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:05.320091 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:05.320112 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:05.320235 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:05.320381 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:26:05.320653 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":121,"threads_started":1}
I20260620 14:26:05.321671 5687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596c80a5e34845b89bcdaa21f048bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41205 } }
I20260620 14:26:05.321988 5687 leader_election.cc:302] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 596c80a5e34845b89bcdaa21f048bd1e; no voters:
I20260620 14:26:05.322283 5687 leader_election.cc:288] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:05.322345 5692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:05.323292 5692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [term 1 LEADER]: Becoming Leader. State: Replica: 596c80a5e34845b89bcdaa21f048bd1e, State: Running, Role: LEADER
I20260620 14:26:05.323469 5687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:05.323777 5692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596c80a5e34845b89bcdaa21f048bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41205 } }
I20260620 14:26:05.324218 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:26:05.326152 5694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 596c80a5e34845b89bcdaa21f048bd1e. Latest consensus state: current_term: 1 leader_uuid: "596c80a5e34845b89bcdaa21f048bd1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596c80a5e34845b89bcdaa21f048bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41205 } } }
I20260620 14:26:05.326267 5694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:05.326236 5693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "596c80a5e34845b89bcdaa21f048bd1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596c80a5e34845b89bcdaa21f048bd1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41205 } } }
I20260620 14:26:05.326339 5693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e [sys.catalog]: This master's current role is: LEADER
W20260620 14:26:05.328474 5706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:26:05.328548 5706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:26:05.328635 5707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:05.329413 5707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:05.334336 5707 catalog_manager.cc:1347] Generated new cluster ID: b3ef0078321645539f7134c31055ff31
I20260620 14:26:05.334400 5707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:05.341034 5707 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:05.342095 5707 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:05.348537 5707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 596c80a5e34845b89bcdaa21f048bd1e: Generated new TSK 0
I20260620 14:26:05.349259 5707 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:26:05.374585 5717 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:26:05.374658 5717 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:26:05.375296 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:05.360210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.0.98.62:41205)
0620 14:26:05.360343 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:05.360347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:05.360519 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:26:05.360661 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:05.360663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:05.360706 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:05.360787 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:26:05.361028 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.361572 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.362740 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.362945 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.363072 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.363359 (+ 287us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:26:05.374912 (+ 11553us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:26:05.375156 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38812: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 14:26:05.375360 5690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38812: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 14:26:05.375365 5716 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:05.360140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41205 (local address 127.0.0.1:38812)
0620 14:26:05.360428 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:05.360442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:05.360556 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:05.360808 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:05.360811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:05.360819 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:26:05.360930 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:05.360936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.361687 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.361690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:05.362639 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:05.362648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.363066 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.363083 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:05.363169 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.363199 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:26:05.375070 (+ 11871us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:26:05.375203 (+ 133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41205: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
I20260620 14:26:05.375900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 5620
2026-06-20T14:26:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:26:05.414758 392 mini_oidc.cc:152] Starting JWKS server
I20260620 14:26:05.415565 392 webserver.cc:460] Webserver started at https://127.0.0.1:38167/ using document root <none> and password file <none>
I20260620 14:26:05.415640 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:26:05.415843 392 webserver.cc:460] Webserver started at http://127.0.0.1:44175/ using document root <none> and password file <none>
2026-06-20T14:26:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:05Z Disabled control of system clock
I20260620 14:26:05.424151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39229
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39229
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38167/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:26:05.527676 5723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:05.527909 5723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:05.527944 5723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:05.531805 5723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:05.531858 5723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:05.531873 5723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:05.531889 5723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:05.531944 5723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:26:05.531966 5723 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:26:05.531993 5723 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38167/jwks.json
I20260620 14:26:05.536392 5723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38167/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:05.537448 5723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:05.538763 5723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:05.547003 5731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:05.546996 5728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:05.546998 5729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:05.547217 5723 server_base.cc:1034] running on GCE node
I20260620 14:26:05.547796 5723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:05.548897 5723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:05.550074 5723 hybrid_clock.cc:648] HybridClock initialized: now 1781965565550025 us; error 61 us; skew 500 ppm
I20260620 14:26:05.551728 5723 webserver.cc:460] Webserver started at http://127.0.98.62:45957/ using document root <none> and password file <none>
I20260620 14:26:05.552325 5723 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:05.552429 5723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:05.552645 5723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:05.554589 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779"
format_stamp: "Formatted at 2026-06-20 14:26:05 on dist-test-slave-pk3j"
server_key: "b5898af10175be8d0d87b828c3be2ae1"
server_key_iv: "a31c4e6bbbde0d9caffa3edce8763214"
server_key_version: "encryptionkey@0"
I20260620 14:26:05.555117 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779"
format_stamp: "Formatted at 2026-06-20 14:26:05 on dist-test-slave-pk3j"
server_key: "b5898af10175be8d0d87b828c3be2ae1"
server_key_iv: "a31c4e6bbbde0d9caffa3edce8763214"
server_key_version: "encryptionkey@0"
I20260620 14:26:05.558526 5723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:26:05.560668 5737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:05.561865 5723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:05.562002 5723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779"
format_stamp: "Formatted at 2026-06-20 14:26:05 on dist-test-slave-pk3j"
server_key: "b5898af10175be8d0d87b828c3be2ae1"
server_key_iv: "a31c4e6bbbde0d9caffa3edce8763214"
server_key_version: "encryptionkey@0"
I20260620 14:26:05.562108 5723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:05.592530 5723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:05.593168 5723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:05.593379 5723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:05.599623 5723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39229
I20260620 14:26:05.599643 5789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39229 every 8 connection(s)
I20260620 14:26:05.600469 5723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:05.603440 5790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:05.608596 5790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779: Bootstrap starting.
I20260620 14:26:05.609302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 5723
I20260620 14:26:05.609412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:05.609702 392 external_mini_cluster.cc:1428] Setting key 9fa3a0db2b5f94a727ad9202e99400cb
I20260620 14:26:05.611326 5790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:05.612054 5790 log.cc:824] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:05.614009 5790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779: No bootstrap required, opened a new log
I20260620 14:26:05.618180 5790 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } }
I20260620 14:26:05.618389 5790 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:05.618445 5790 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2e09dcb3bd4c449ab28aa1c01d9779, State: Initialized, Role: FOLLOWER
I20260620 14:26:05.618759 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:05.610872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.0.98.62:39229)
0620 14:26:05.611491 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:05.611509 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:05.611544 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:26:05.612206 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:05.612223 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:05.612707 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:05.613047 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:05.613416 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.614323 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.615403 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.615980 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.616144 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.616934 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:05.616939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:05.616951 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:05.617396 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:05.617447 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:05.617465 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:05.617554 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:05.617692 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:26:05.617968 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":124,"threads_started":1}
I20260620 14:26:05.618921 5790 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } }
I20260620 14:26:05.619086 5790 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:05.619153 5790 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:05.619242 5790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:05.620052 5790 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } }
I20260620 14:26:05.620369 5790 leader_election.cc:302] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb2e09dcb3bd4c449ab28aa1c01d9779; no voters:
I20260620 14:26:05.620704 5790 leader_election.cc:288] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:05.620798 5795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:05.621045 5795 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [term 1 LEADER]: Becoming Leader. State: Replica: eb2e09dcb3bd4c449ab28aa1c01d9779, State: Running, Role: LEADER
I20260620 14:26:05.621520 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:26:05.621665 5795 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } }
I20260620 14:26:05.621984 5790 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:05.622673 5795 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb2e09dcb3bd4c449ab28aa1c01d9779. Latest consensus state: current_term: 1 leader_uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } } }
I20260620 14:26:05.622764 5795 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:05.623216 5802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:05.623562 5796 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb2e09dcb3bd4c449ab28aa1c01d9779" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39229 } } }
I20260620 14:26:05.623677 5796 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:05.626238 5802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:05.630980 5802 catalog_manager.cc:1347] Generated new cluster ID: 70887d4a6d4c4127a29eb01ac1425bd7
I20260620 14:26:05.631040 5802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:05.650557 5802 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:05.651531 5802 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:05.655851 5802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb2e09dcb3bd4c449ab28aa1c01d9779: Generated new TSK 0
I20260620 14:26:05.656478 5802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:26:05.671111 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:05.666245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40630 (local address 127.0.98.62:39229)
0620 14:26:05.666349 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:05.666354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:05.666551 (+ 197us) server_negotiation.cc:407] Connection header received
0620 14:26:05.666673 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:05.666676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:05.666731 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:05.666840 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:26:05.667039 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.667833 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.668644 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.668989 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.669086 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.669373 (+ 287us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:26:05.670766 (+ 1393us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:26:05.670987 (+ 221us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40630: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 14:26:05.671173 5793 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40630: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:26:05.672904 5819 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:05.666126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39229 (local address 127.0.0.1:40630)
0620 14:26:05.666469 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:05.666480 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:05.666574 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:05.666849 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:05.666851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:05.666856 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:26:05.666944 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:05.666948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.667941 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.667944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:05.668537 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:05.668545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.669092 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.669108 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:05.669169 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.669190 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:26:05.670916 (+ 1726us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:26:05.672737 (+ 1821us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1574,"thread_start_us":108,"threads_started":1}
I20260620 14:26:05.673532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 5723
E20260620 14:26:05.681644 5820 webserver.cc:533] Webserver: sslize failed:
E20260620 14:26:05.681718 5820 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T14:26:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (296 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:26:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:05Z Disabled control of system clock
I20260620 14:26:05.719045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36401
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36401
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/testchainca.cert with env {}
W20260620 14:26:05.821072 5824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:05.821347 5824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:05.821388 5824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:05.825213 5824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:05.825258 5824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:05.825273 5824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:05.825289 5824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:05.825307 5824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:26:05.825322 5824 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:26:05.825335 5824 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:26:05.830250 5824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:05.830999 5824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:05.831805 5824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:05.839365 5830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:05.839401 5832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:05.839372 5829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:05.839519 5824 server_base.cc:1034] running on GCE node
I20260620 14:26:05.840152 5824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:05.840986 5824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:05.842139 5824 hybrid_clock.cc:648] HybridClock initialized: now 1781965565842119 us; error 32 us; skew 500 ppm
I20260620 14:26:05.843724 5824 webserver.cc:460] Webserver started at http://127.0.98.62:33545/ using document root <none> and password file <none>
I20260620 14:26:05.844147 5824 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:05.844197 5824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:05.844334 5824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:05.846102 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "5507194de99f4e61a936ad0fc15c3c3b"
format_stamp: "Formatted at 2026-06-20 14:26:05 on dist-test-slave-pk3j"
server_key: "921b89b83682e020a67793565147b08f"
server_key_iv: "a32c1cb1aa056954ca680e8deec34c48"
server_key_version: "encryptionkey@0"
I20260620 14:26:05.846588 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "5507194de99f4e61a936ad0fc15c3c3b"
format_stamp: "Formatted at 2026-06-20 14:26:05 on dist-test-slave-pk3j"
server_key: "921b89b83682e020a67793565147b08f"
server_key_iv: "a32c1cb1aa056954ca680e8deec34c48"
server_key_version: "encryptionkey@0"
I20260620 14:26:05.849720 5824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:26:05.851747 5838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:05.852799 5824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:05.852962 5824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "5507194de99f4e61a936ad0fc15c3c3b"
format_stamp: "Formatted at 2026-06-20 14:26:05 on dist-test-slave-pk3j"
server_key: "921b89b83682e020a67793565147b08f"
server_key_iv: "a32c1cb1aa056954ca680e8deec34c48"
server_key_version: "encryptionkey@0"
I20260620 14:26:05.853065 5824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:05.866079 5824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:05.866631 5824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:05.866811 5824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:05.872905 5824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36401
I20260620 14:26:05.872933 5890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36401 every 8 connection(s)
I20260620 14:26:05.873755 5824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:05.874085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 5824
I20260620 14:26:05.874194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:05.874533 392 external_mini_cluster.cc:1428] Setting key b831a3921ca8ca0a8c5db97c7b6d9aa5
I20260620 14:26:05.876904 5891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:05.881829 5891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b: Bootstrap starting.
I20260620 14:26:05.882776 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:05.875748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.0.98.62:36401)
0620 14:26:05.876331 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:05.876346 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:05.876376 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:26:05.876937 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:05.876984 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:05.877456 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:05.877757 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:05.878127 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.879008 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.879862 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.880285 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.880463 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.881275 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:05.881279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:05.881290 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:05.881618 (+ 328us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:05.881665 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:05.881683 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:05.881787 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:05.881929 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:26:05.882183 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":138,"threads_started":1}
I20260620 14:26:05.884019 5891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:05.884811 5891 log.cc:824] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:05.885385 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:26:05.886910 5891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b: No bootstrap required, opened a new log
I20260620 14:26:05.890196 5891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5507194de99f4e61a936ad0fc15c3c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36401 } }
I20260620 14:26:05.890379 5891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:05.890462 5891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5507194de99f4e61a936ad0fc15c3c3b, State: Initialized, Role: FOLLOWER
I20260620 14:26:05.890844 5891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5507194de99f4e61a936ad0fc15c3c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36401 } }
I20260620 14:26:05.890983 5891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:05.891067 5891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:05.891177 5891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:05.892007 5891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5507194de99f4e61a936ad0fc15c3c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36401 } }
I20260620 14:26:05.892313 5891 leader_election.cc:302] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5507194de99f4e61a936ad0fc15c3c3b; no voters:
I20260620 14:26:05.892586 5891 leader_election.cc:288] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:05.892691 5896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:05.892941 5896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [term 1 LEADER]: Becoming Leader. State: Replica: 5507194de99f4e61a936ad0fc15c3c3b, State: Running, Role: LEADER
I20260620 14:26:05.893276 5896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5507194de99f4e61a936ad0fc15c3c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36401 } }
I20260620 14:26:05.893745 5891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:05.894804 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5507194de99f4e61a936ad0fc15c3c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5507194de99f4e61a936ad0fc15c3c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36401 } } }
I20260620 14:26:05.894935 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:05.895177 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5507194de99f4e61a936ad0fc15c3c3b. Latest consensus state: current_term: 1 leader_uuid: "5507194de99f4e61a936ad0fc15c3c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5507194de99f4e61a936ad0fc15c3c3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36401 } } }
I20260620 14:26:05.895265 5905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:05.895359 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:05.898231 5905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:05.903009 5905 catalog_manager.cc:1347] Generated new cluster ID: 5f45f902fdbe4619b2c569bc6cc7c442
I20260620 14:26:05.903066 5905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:05.927188 5905 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:05.928229 5905 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:05.937572 5905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5507194de99f4e61a936ad0fc15c3c3b: Generated new TSK 0
I20260620 14:26:05.938124 5905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:26:06.011968 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:05.955361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.0.98.62:36401)
0620 14:26:05.955486 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:05.955491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:05.955687 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:26:05.955881 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:05.955883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:05.955924 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:05.956003 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:26:05.956256 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.956789 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.958009 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.958199 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.958274 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.958564 (+ 290us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:26:06.011489 (+ 52925us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:26:06.011826 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54654: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 14:26:06.012053 5894 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54654: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:26:06.012079 5920 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:05.955322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36401 (local address 127.0.0.1:54654)
0620 14:26:05.955599 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:05.955613 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:05.955719 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:05.956026 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:05.956029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:05.956036 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:26:05.956158 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:05.956164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.956896 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.956900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:05.957906 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:05.957916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:05.958289 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:05.958309 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:05.958392 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:05.958418 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:26:06.011706 (+ 53288us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:26:06.011908 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36401: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":113,"threads_started":1}
I20260620 14:26:06.012720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 5824
2026-06-20T14:26:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (320 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (110 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (110 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:26:06 dist-test-slave-pk3j krb5kdc[5932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:26:06 dist-test-slave-pk3j krb5kdc[5932](info): set up 2 sockets
Jun 20 14:26:06 dist-test-slave-pk3j krb5kdc[5932](info): commencing operation
krb5kdc: starting...
W20260620 14:26:07.233690 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:26:07 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:26:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:26:07.337503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42015
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:07.435825 5945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:07.436020 5945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:07.436053 5945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:07.439873 5945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:07.439915 5945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:07.439932 5945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:07.439946 5945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:07.439960 5945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:07.444625 5945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:07.445490 5945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:07.446362 5945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:07.453838 5951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:07.453830 5950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:07.453845 5953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:07.454137 5945 server_base.cc:1034] running on GCE node
I20260620 14:26:07.454531 5945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:07.455430 5945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:07.456579 5945 hybrid_clock.cc:648] HybridClock initialized: now 1781965567456563 us; error 28 us; skew 500 ppm
Jun 20 14:26:07 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:07.459076 5945 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:26:07.459800 5945 webserver.cc:460] Webserver started at http://127.0.98.62:33387/ using document root <none> and password file <none>
I20260620 14:26:07.460278 5945 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:07.460359 5945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:07.460564 5945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:07.462430 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "6a7ea393ebfc412f8d9a989402dbd9bc"
format_stamp: "Formatted at 2026-06-20 14:26:07 on dist-test-slave-pk3j"
server_key: "eb832d4dc5cad590d6100cfc2b41a636"
server_key_iv: "c719862b264b4923fb56a07c7bfb1b42"
server_key_version: "encryptionkey@0"
I20260620 14:26:07.462934 5945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a7ea393ebfc412f8d9a989402dbd9bc"
format_stamp: "Formatted at 2026-06-20 14:26:07 on dist-test-slave-pk3j"
server_key: "eb832d4dc5cad590d6100cfc2b41a636"
server_key_iv: "c719862b264b4923fb56a07c7bfb1b42"
server_key_version: "encryptionkey@0"
I20260620 14:26:07.466267 5945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:26:07.468333 5960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:07.469411 5945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:26:07.469532 5945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "6a7ea393ebfc412f8d9a989402dbd9bc"
format_stamp: "Formatted at 2026-06-20 14:26:07 on dist-test-slave-pk3j"
server_key: "eb832d4dc5cad590d6100cfc2b41a636"
server_key_iv: "c719862b264b4923fb56a07c7bfb1b42"
server_key_version: "encryptionkey@0"
I20260620 14:26:07.469628 5945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:07.495339 5945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:07.498396 5945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:07.498570 5945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:07.504736 5945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42015
I20260620 14:26:07.504981 6012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42015 every 8 connection(s)
I20260620 14:26:07.505707 5945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:07.508370 6013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:07.512399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 5945
I20260620 14:26:07.512537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:07.512835 392 external_mini_cluster.cc:1428] Setting key c1a90767efe0ffbafc3a26d6016b8c1c
I20260620 14:26:07.513545 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc: Bootstrap starting.
I20260620 14:26:07.515653 6013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:07.516355 6013 log.cc:824] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:07.518080 6013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc: No bootstrap required, opened a new log
Jun 20 14:26:07 dist-test-slave-pk3j krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:07.521657 6013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7ea393ebfc412f8d9a989402dbd9bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260620 14:26:07.521836 6013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:07.521891 6013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7ea393ebfc412f8d9a989402dbd9bc, State: Initialized, Role: FOLLOWER
I20260620 14:26:07.522310 6013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7ea393ebfc412f8d9a989402dbd9bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260620 14:26:07.522433 6013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:07.522533 6013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:07.522634 6013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:07.523379 6013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7ea393ebfc412f8d9a989402dbd9bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260620 14:26:07.523687 6013 leader_election.cc:302] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a7ea393ebfc412f8d9a989402dbd9bc; no voters:
I20260620 14:26:07.523974 6013 leader_election.cc:288] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:07.524077 6018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:07.524295 6018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [term 1 LEADER]: Becoming Leader. State: Replica: 6a7ea393ebfc412f8d9a989402dbd9bc, State: Running, Role: LEADER
I20260620 14:26:07.524608 6018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7ea393ebfc412f8d9a989402dbd9bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260620 14:26:07.525156 6013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:07.526163 6020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a7ea393ebfc412f8d9a989402dbd9bc. Latest consensus state: current_term: 1 leader_uuid: "6a7ea393ebfc412f8d9a989402dbd9bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7ea393ebfc412f8d9a989402dbd9bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } } }
I20260620 14:26:07.526319 6020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:07.526450 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:07.514114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.0.98.62:42015)
0620 14:26:07.514490 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:07.514502 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:07.514532 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:26:07.515074 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:07.515088 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:07.515381 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:07.515683 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:07.516447 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:07.517673 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:07.518523 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:07.518962 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:07.519127 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:07.521661 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:07.521665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:07.521677 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:07.521699 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:07.523380 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:07.523823 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:07.523828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:07.523833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:07.523944 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:07.524328 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:07.524331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:07.524332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:07.524742 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:07.524930 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:07.525196 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 14:26:07.525484 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260620 14:26:07.526638 6019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a7ea393ebfc412f8d9a989402dbd9bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7ea393ebfc412f8d9a989402dbd9bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } } }
I20260620 14:26:07.526726 6019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:07.527513 6027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:07.530390 6027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:07.535041 6027 catalog_manager.cc:1347] Generated new cluster ID: d1313aab0aa3461697b8131927f56475
I20260620 14:26:07.535097 6027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:07.570451 6027 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:07.571293 6027 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:07.583535 6027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a7ea393ebfc412f8d9a989402dbd9bc: Generated new TSK 0
I20260620 14:26:07.584121 6027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:26:07.615764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:07.717123 6039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:07.717418 6039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:07.717458 6039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:07.721340 6039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:07.721390 6039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:07.721457 6039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:07.726415 6039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:07.727258 6039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:07.728132 6039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:07.736795 6045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:07.736863 6044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:07.736863 6047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:07.737098 6039 server_base.cc:1034] running on GCE node
I20260620 14:26:07.737543 6039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:07.738081 6039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:07.739260 6039 hybrid_clock.cc:648] HybridClock initialized: now 1781965567739248 us; error 34 us; skew 500 ppm
Jun 20 14:26:07 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:07.741882 6039 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:26:07.742660 6039 webserver.cc:460] Webserver started at http://127.0.98.1:43757/ using document root <none> and password file <none>
I20260620 14:26:07.743180 6039 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:07.743268 6039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:07.743505 6039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:07.745314 6039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "25966acaca7843e7a856fb1244ce806d"
format_stamp: "Formatted at 2026-06-20 14:26:07 on dist-test-slave-pk3j"
server_key: "340b6c86cff4c56613803fe09e43caa1"
server_key_iv: "e83e75f0bde2b551e6df488575607ed9"
server_key_version: "encryptionkey@0"
I20260620 14:26:07.745821 6039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "25966acaca7843e7a856fb1244ce806d"
format_stamp: "Formatted at 2026-06-20 14:26:07 on dist-test-slave-pk3j"
server_key: "340b6c86cff4c56613803fe09e43caa1"
server_key_iv: "e83e75f0bde2b551e6df488575607ed9"
server_key_version: "encryptionkey@0"
I20260620 14:26:07.749063 6039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:26:07.751271 6054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:07.752367 6039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:07.752494 6039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "25966acaca7843e7a856fb1244ce806d"
format_stamp: "Formatted at 2026-06-20 14:26:07 on dist-test-slave-pk3j"
server_key: "340b6c86cff4c56613803fe09e43caa1"
server_key_iv: "e83e75f0bde2b551e6df488575607ed9"
server_key_version: "encryptionkey@0"
I20260620 14:26:07.752597 6039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:07.763172 6039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:07.766142 6039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:07.766323 6039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:07.766820 6039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:07.767529 6039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:07.767597 6039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:07.767663 6039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:07.767712 6039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:07.776439 6039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45169
I20260620 14:26:07.776477 6167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45169 every 8 connection(s)
I20260620 14:26:07.777244 6039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:26:07.780663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 6039
I20260620 14:26:07.780805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:26:07.781050 392 external_mini_cluster.cc:1428] Setting key 1e2146ace5deef4c39aa15cab469e08b
Jun 20 14:26:07 dist-test-slave-pk3j krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:07.789902 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:07.778761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55083 (local address 127.0.98.62:42015)
0620 14:26:07.778875 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:07.778884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:07.779534 (+ 650us) server_negotiation.cc:407] Connection header received
0620 14:26:07.780827 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:07.780830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:07.780897 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:07.780992 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:07.782074 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:07.782565 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:07.783645 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:07.783894 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:07.784002 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:07.786572 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:07.786575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:07.786577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:07.786598 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:07.788001 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:07.788534 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:07.788536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:07.788537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:07.788602 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:07.788953 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:07.788955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:07.788955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:07.789088 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:07.789154 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:07.789716 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 14:26:07.789799 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:26:07.790743 6170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:07.779015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42015 (local address 127.0.98.1:55083)
0620 14:26:07.779408 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:07.779443 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:07.780627 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:07.781172 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:07.781180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:07.781542 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:07.781936 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:07.781948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:07.782683 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:07.782686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:07.783523 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:07.783531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:07.784032 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:07.784036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:07.784189 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:07.784709 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:07.784730 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:07.786413 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:07.788110 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:07.788118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:07.788132 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:07.788445 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:07.788688 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:07.788692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:07.788696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:07.788854 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:07.789185 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:07.789192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:07.789485 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 14:26:07.789691 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 14:26:07.789976 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":107,"threads_started":1}
I20260620 14:26:07.791744 6168 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42015
I20260620 14:26:07.792002 6168 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:26:07.792425 6168 heartbeater.cc:511] Master 127.0.98.62:42015 requested a full tablet report, sending...
I20260620 14:26:07.793850 5977 ts_manager.cc:194] Registered new tserver with Master: 25966acaca7843e7a856fb1244ce806d (127.0.98.1:45169)
I20260620 14:26:07.795071 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:26:07.805588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:07.907627 6173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:07.907831 6173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:07.907869 6173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:07.911767 6173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:07.911820 6173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:07.911917 6173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:07.916776 6173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:07.917725 6173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:07.918613 6173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:07.927330 6179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:07.927277 6178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:07.927271 6181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:07.927587 6173 server_base.cc:1034] running on GCE node
I20260620 14:26:07.928013 6173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:07.928551 6173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:07.929701 6173 hybrid_clock.cc:648] HybridClock initialized: now 1781965567929691 us; error 23 us; skew 500 ppm
Jun 20 14:26:07 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:07.932153 6173 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:26:07.932914 6173 webserver.cc:460] Webserver started at http://127.0.98.2:39949/ using document root <none> and password file <none>
I20260620 14:26:07.933555 6173 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:07.933655 6173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:07.933841 6173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:07.935549 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "80e667c92c2c4d26b397c12643b464b4"
format_stamp: "Formatted at 2026-06-20 14:26:07 on dist-test-slave-pk3j"
server_key: "7803d9855e6c74d0f68798d5e0aea81f"
server_key_iv: "91fb4e6fffb594280cbc2c9b23a232da"
server_key_version: "encryptionkey@0"
I20260620 14:26:07.936046 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80e667c92c2c4d26b397c12643b464b4"
format_stamp: "Formatted at 2026-06-20 14:26:07 on dist-test-slave-pk3j"
server_key: "7803d9855e6c74d0f68798d5e0aea81f"
server_key_iv: "91fb4e6fffb594280cbc2c9b23a232da"
server_key_version: "encryptionkey@0"
I20260620 14:26:07.939810 6173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:26:07.942973 6188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:07.944393 6173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:26:07.944501 6173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "80e667c92c2c4d26b397c12643b464b4"
format_stamp: "Formatted at 2026-06-20 14:26:07 on dist-test-slave-pk3j"
server_key: "7803d9855e6c74d0f68798d5e0aea81f"
server_key_iv: "91fb4e6fffb594280cbc2c9b23a232da"
server_key_version: "encryptionkey@0"
I20260620 14:26:07.944599 6173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:07.966593 6173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:07.969074 6173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:07.969267 6173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:07.969776 6173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:07.970487 6173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:07.970531 6173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:07.970593 6173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:07.970631 6173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:07.979063 6173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44271
I20260620 14:26:07.979084 6301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44271 every 8 connection(s)
I20260620 14:26:07.979799 6173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:07.980388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 6173
I20260620 14:26:07.980530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:26:07.980798 392 external_mini_cluster.cc:1428] Setting key 5229f3af74465efadcadb2ffca848235
Jun 20 14:26:07 dist-test-slave-pk3j krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:26:07.993623 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:07.981655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36613 (local address 127.0.98.62:42015)
0620 14:26:07.981796 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:07.981800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:07.982525 (+ 725us) server_negotiation.cc:407] Connection header received
0620 14:26:07.983991 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:07.983993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:07.984032 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:07.984097 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:07.985198 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:07.985721 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:07.987464 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:07.987874 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:07.988003 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:07.990407 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:07.990410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:07.990411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:07.990431 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:07.991770 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:07.992204 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:07.992206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:07.992207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:07.992273 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:07.992646 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:07.992648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:07.992649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:07.992802 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:07.992881 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:07.993440 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 14:26:07.993520 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:26:07.994640 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:07.981905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42015 (local address 127.0.98.2:36613)
0620 14:26:07.982380 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:07.982425 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:07.983784 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:07.984267 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:07.984277 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:07.984686 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:07.985089 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:07.985100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:07.985839 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:07.985845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:07.987330 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:07.987340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:07.988010 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:07.988012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:07.988145 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:07.988681 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:07.988694 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:07.990253 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:07.991881 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:07.991889 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:07.991899 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:07.992123 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:07.992348 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:07.992350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:07.992351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:07.992569 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:07.992950 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:07.992959 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:07.993185 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 14:26:07.993405 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 14:26:07.993712 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":148,"threads_started":1}
I20260620 14:26:07.995700 6302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42015
I20260620 14:26:07.995920 6302 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:07.996419 6302 heartbeater.cc:511] Master 127.0.98.62:42015 requested a full tablet report, sending...
I20260620 14:26:07.997505 5977 ts_manager.cc:194] Registered new tserver with Master: 80e667c92c2c4d26b397c12643b464b4 (127.0.98.2:44271)
I20260620 14:26:07.998029 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:26:08.007463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:08.107931 6307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:08.108160 6307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:08.108230 6307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:08.112099 6307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:08.112174 6307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:08.112254 6307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:08.116918 6307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:08.117796 6307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:08.118664 6307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:08.127007 6312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:08.127003 6313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:08.127056 6315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:08.127342 6307 server_base.cc:1034] running on GCE node
I20260620 14:26:08.127744 6307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:08.128260 6307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:08.129436 6307 hybrid_clock.cc:648] HybridClock initialized: now 1781965568129415 us; error 39 us; skew 500 ppm
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:08.131970 6307 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:26:08.132733 6307 webserver.cc:460] Webserver started at http://127.0.98.3:41275/ using document root <none> and password file <none>
I20260620 14:26:08.133261 6307 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:08.133371 6307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:08.133652 6307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:08.135349 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "79aa9255cac44c59b435fc81a683808b"
format_stamp: "Formatted at 2026-06-20 14:26:08 on dist-test-slave-pk3j"
server_key: "0b91d8f7dc397a8d8baec46bda5b68d9"
server_key_iv: "7378775b3cd2b6b728b18e455500a518"
server_key_version: "encryptionkey@0"
I20260620 14:26:08.135859 6307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "79aa9255cac44c59b435fc81a683808b"
format_stamp: "Formatted at 2026-06-20 14:26:08 on dist-test-slave-pk3j"
server_key: "0b91d8f7dc397a8d8baec46bda5b68d9"
server_key_iv: "7378775b3cd2b6b728b18e455500a518"
server_key_version: "encryptionkey@0"
I20260620 14:26:08.139117 6307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:08.141304 6322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:08.142375 6307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:08.142485 6307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "79aa9255cac44c59b435fc81a683808b"
format_stamp: "Formatted at 2026-06-20 14:26:08 on dist-test-slave-pk3j"
server_key: "0b91d8f7dc397a8d8baec46bda5b68d9"
server_key_iv: "7378775b3cd2b6b728b18e455500a518"
server_key_version: "encryptionkey@0"
I20260620 14:26:08.142575 6307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:08.167765 6307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:08.170620 6307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:08.170794 6307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:08.171267 6307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:08.171968 6307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:08.172042 6307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:08.172142 6307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:08.172194 6307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:08.180286 6307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37023
I20260620 14:26:08.180315 6435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37023 every 8 connection(s)
I20260620 14:26:08.181006 6307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:08.182933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 6307
I20260620 14:26:08.183039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:26:08.183301 392 external_mini_cluster.cc:1428] Setting key 21bbf2ddf61350a7a184ee41f07142f3
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:08.192963 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:08.182503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44377 (local address 127.0.98.62:42015)
0620 14:26:08.182615 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:08.182618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:08.183398 (+ 780us) server_negotiation.cc:407] Connection header received
0620 14:26:08.184832 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:08.184835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:08.184886 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:08.184964 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:08.185981 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:08.186507 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:08.187602 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:08.187823 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:08.187928 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:08.190001 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:08.190005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:08.190007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:08.190030 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:08.191169 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:08.191621 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:08.191625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:08.191627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:08.191716 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:08.192029 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:08.192034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:08.192035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:08.192195 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:08.192292 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:08.192706 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 14:26:08.192808 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:26:08.193646 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:08.182829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42015 (local address 127.0.98.3:44377)
0620 14:26:08.183269 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:08.183303 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:08.184648 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:08.185105 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:08.185112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:08.185460 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:08.185837 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:08.185848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:08.186640 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:08.186642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:08.187479 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:08.187488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:08.187955 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:08.187958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:08.188113 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:08.188538 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:08.188552 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:08.189859 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:08.191274 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:08.191279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:08.191290 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:08.191521 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:08.191806 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:08.191810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:08.191812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:08.191938 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:08.192301 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:08.192309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:08.192504 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 14:26:08.192666 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:26:08.192932 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":121,"threads_started":1}
I20260620 14:26:08.194525 6436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42015
I20260620 14:26:08.194725 6436 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:08.195119 6436 heartbeater.cc:511] Master 127.0.98.62:42015 requested a full tablet report, sending...
I20260620 14:26:08.196087 5977 ts_manager.cc:194] Registered new tserver with Master: 79aa9255cac44c59b435fc81a683808b (127.0.98.3:37023)
I20260620 14:26:08.196704 5977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44377
I20260620 14:26:08.196794 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/test-admin.keytab.
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:08.210230 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:26:08.212443 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:08.211242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.0.98.1:45169)
0620 14:26:08.211511 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:08.211519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:08.211539 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:26:08.211579 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:08.211582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:08.211760 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:08.211863 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:08.212247 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46164: BlockingRecv error: recv got EOF from 127.0.0.1:46164 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
W20260620 14:26:08.212499 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:08.213459 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:08.214282 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:26:08.222875 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:08.214991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.0.98.1:45169)
0620 14:26:08.215088 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:08.215091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:08.215133 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:26:08.215229 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:08.215230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:08.215259 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:08.215335 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:08.215802 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:08.216519 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:08.217320 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:08.217645 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:08.217722 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:08.219627 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:08.219635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:08.219643 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:08.219663 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:08.221190 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:08.221661 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:08.221666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:08.221667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:08.221737 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:08.222007 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:08.222011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:08.222013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:08.222212 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:08.222451 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:08.222617 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:26:08.222742 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 14:26:08.226886 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:08.225706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.0.98.1:45169)
0620 14:26:08.225823 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:08.225826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:08.226083 (+ 257us) server_negotiation.cc:407] Connection header received
0620 14:26:08.226204 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:08.226208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:08.226261 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:08.226376 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:08.226801 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46188: BlockingRecv error: recv got EOF from 127.0.0.1:46188 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:26:08.227018 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:08.227957 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:08.228735 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:26:08.235992 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:08.229441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46192 (local address 127.0.98.1:45169)
0620 14:26:08.229536 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:08.229538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:08.229653 (+ 115us) server_negotiation.cc:407] Connection header received
0620 14:26:08.229740 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:08.229742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:08.229769 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:08.229823 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:08.230199 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:08.230650 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:08.231421 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:08.231618 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:08.231704 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:08.233758 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:08.233760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:08.233762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:08.233781 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:08.234866 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:08.235195 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:08.235197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:08.235198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:08.235257 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:08.235466 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:08.235468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:08.235469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:08.235587 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:08.235671 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:08.235830 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:26:08.235909 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 14:26:08.238175 6443 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:08.237406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.0.98.1:45169)
0620 14:26:08.237549 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:08.237553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:08.237605 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:26:08.237684 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:08.237686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:08.237713 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:08.237770 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:08.238100 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46204: BlockingRecv error: recv got EOF from 127.0.0.1:46204 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 14:26:08.238262 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781965536974027-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:08.239079 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:08.239836 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[5932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:26:08.247179 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:08.240481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.0.98.1:45169)
0620 14:26:08.240589 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:08.240593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:08.240689 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:26:08.240813 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:08.240816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:08.240859 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:08.240934 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:08.241419 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:08.241825 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:08.242508 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:08.242693 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:08.242786 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:08.244565 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:08.244569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:08.244571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:08.244595 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:08.245733 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:08.246085 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:08.246090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:08.246091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:08.246155 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:08.246450 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:08.246455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:08.246457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:08.246591 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:08.246708 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:08.246927 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 14:26:08.247023 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:26:08.247941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 6039
I20260620 14:26:08.253094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 6173
I20260620 14:26:08.258754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 6307
I20260620 14:26:08.265120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 5945
2026-06-20T14:26:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2032 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[6456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[6456](info): set up 2 sockets
Jun 20 14:26:08 dist-test-slave-pk3j krb5kdc[6456](info): commencing operation
krb5kdc: starting...
W20260620 14:26:09.265367 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:26:09 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:26:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:26:09.369176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42021
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:09.470050 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:09.470259 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:09.470295 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:09.474064 6469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:09.474108 6469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:09.474128 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:09.474177 6469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:09.474205 6469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:09.478969 6469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:09.479785 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:09.480645 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:09.488296 6474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:09.488281 6475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:09.488346 6469 server_base.cc:1034] running on GCE node
W20260620 14:26:09.488282 6477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:09.489005 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:09.489974 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:09.491128 6469 hybrid_clock.cc:648] HybridClock initialized: now 1781965569491112 us; error 28 us; skew 500 ppm
Jun 20 14:26:09 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:09.493467 6469 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:26:09.494170 6469 webserver.cc:460] Webserver started at http://127.0.98.62:41621/ using document root <none> and password file <none>
I20260620 14:26:09.494628 6469 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:09.494709 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:09.494901 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:09.496603 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "9ef79ad2a6de42ecacfb6489b5381f63"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "07f1cd12838d503fa1959026c3f06335"
server_key_iv: "bd00c237df780248916abff95f181868"
server_key_version: "encryptionkey@0"
I20260620 14:26:09.497117 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ef79ad2a6de42ecacfb6489b5381f63"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "07f1cd12838d503fa1959026c3f06335"
server_key_iv: "bd00c237df780248916abff95f181868"
server_key_version: "encryptionkey@0"
I20260620 14:26:09.500362 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:26:09.502434 6484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:09.503469 6469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:09.503595 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "9ef79ad2a6de42ecacfb6489b5381f63"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "07f1cd12838d503fa1959026c3f06335"
server_key_iv: "bd00c237df780248916abff95f181868"
server_key_version: "encryptionkey@0"
I20260620 14:26:09.503693 6469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:09.514446 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:09.516755 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:09.516937 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:09.522897 6469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42021
I20260620 14:26:09.523141 6536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42021 every 8 connection(s)
I20260620 14:26:09.523723 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:09.526177 6537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:09.531040 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63: Bootstrap starting.
I20260620 14:26:09.533169 6537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:09.533665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 6469
I20260620 14:26:09.533802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:09.533969 6537 log.cc:824] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:09.534077 392 external_mini_cluster.cc:1428] Setting key 2ddbe738a9a77a158bbfba0ce9da491f
I20260620 14:26:09.535888 6537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63: No bootstrap required, opened a new log
I20260620 14:26:09.540051 6537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 14:26:09.540269 6537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:09.540385 6537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef79ad2a6de42ecacfb6489b5381f63, State: Initialized, Role: FOLLOWER
I20260620 14:26:09.540846 6537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 14:26:09.541002 6537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:09.541074 6537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:09.541180 6537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:09.542249 6537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
Jun 20 14:26:09 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:09.542589 6537 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef79ad2a6de42ecacfb6489b5381f63; no voters:
I20260620 14:26:09.542876 6537 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:09.542975 6542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:09.543241 6542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 1 LEADER]: Becoming Leader. State: Replica: 9ef79ad2a6de42ecacfb6489b5381f63, State: Running, Role: LEADER
I20260620 14:26:09.543591 6542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260620 14:26:09.544111 6537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:09.544723 6542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef79ad2a6de42ecacfb6489b5381f63. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } }
I20260620 14:26:09.544862 6542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:09.545135 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } }
I20260620 14:26:09.545235 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:09.545270 6550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:09.548396 6550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:09.548657 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:09.535172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.0.98.62:42021)
0620 14:26:09.535501 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:09.535523 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:09.535554 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:26:09.536170 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:09.536187 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:09.536502 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:09.536756 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:09.537709 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:09.538630 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:09.539827 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:09.540241 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:09.540426 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:09.543119 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:09.543125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:09.543139 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:09.543167 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:09.545271 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:09.545738 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:09.545744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:09.545750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:09.545888 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:09.546191 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:09.546195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:09.546197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:09.546670 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:09.546868 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:09.547540 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 14:26:09.547817 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260620 14:26:09.553750 6550 catalog_manager.cc:1347] Generated new cluster ID: 6c8df4cacf1b4e0d8cc811934d63ba7c
I20260620 14:26:09.553812 6550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:09.565460 6550 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:09.566324 6550 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:09.569442 6550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63: Generated new TSK 0
I20260620 14:26:09.570025 6550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:26:09.595510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:09.699755 6563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:09.700018 6563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:09.700085 6563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:09.703928 6563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:09.704005 6563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:09.704087 6563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:09.709264 6563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:09.710246 6563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:09.711201 6563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:09.720050 6569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:09.720045 6568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:09.720199 6563 server_base.cc:1034] running on GCE node
W20260620 14:26:09.720321 6571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:09.720659 6563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:09.721247 6563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:09.722436 6563 hybrid_clock.cc:648] HybridClock initialized: now 1781965569722429 us; error 27 us; skew 500 ppm
Jun 20 14:26:09 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:09.728628 6563 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:26:09.729523 6563 webserver.cc:460] Webserver started at http://127.0.98.1:34783/ using document root <none> and password file <none>
I20260620 14:26:09.730046 6563 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:09.730130 6563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:09.730317 6563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:09.732168 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "0452603959bd46158c77c4f7b2f2ddef"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "ea9c39aca30cf799805e7c8c102e5bd5"
server_key_iv: "9eafda8a59759f1b79bcc2ceb4a5248c"
server_key_version: "encryptionkey@0"
I20260620 14:26:09.732735 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0452603959bd46158c77c4f7b2f2ddef"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "ea9c39aca30cf799805e7c8c102e5bd5"
server_key_iv: "9eafda8a59759f1b79bcc2ceb4a5248c"
server_key_version: "encryptionkey@0"
I20260620 14:26:09.736156 6563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:26:09.738337 6578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:09.739351 6563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:09.739473 6563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "0452603959bd46158c77c4f7b2f2ddef"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "ea9c39aca30cf799805e7c8c102e5bd5"
server_key_iv: "9eafda8a59759f1b79bcc2ceb4a5248c"
server_key_version: "encryptionkey@0"
I20260620 14:26:09.739585 6563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:09.760140 6563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:09.763135 6563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:09.763342 6563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:09.763839 6563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:09.764544 6563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:09.764623 6563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:09.764686 6563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:09.764731 6563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:09.773038 6563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45309
I20260620 14:26:09.773061 6691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45309 every 8 connection(s)
I20260620 14:26:09.773752 6563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:26:09.780973 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 6563
I20260620 14:26:09.781092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:26:09.781416 392 external_mini_cluster.cc:1428] Setting key c0b613868926ddb3aa7456a63a0471ff
Jun 20 14:26:09 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:09.786311 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:09.775272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54261 (local address 127.0.98.62:42021)
0620 14:26:09.775419 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:09.775423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:09.776519 (+ 1096us) server_negotiation.cc:407] Connection header received
0620 14:26:09.777358 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:09.777361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:09.777415 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:09.777518 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:09.778506 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:09.779016 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:09.780039 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:09.780268 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:09.780371 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:09.782751 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:09.782756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:09.782759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:09.782784 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:09.784366 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:09.784859 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:09.784863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:09.784865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:09.784945 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:09.785314 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:09.785320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:09.785322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:09.785493 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:09.785595 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:09.786025 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 14:26:09.786135 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:26:09.786958 6694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:09.775606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:54261)
0620 14:26:09.776390 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:09.776425 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:09.777149 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:09.777680 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:09.777686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:09.777972 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:09.778370 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:09.778381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:09.779150 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:09.779152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:09.779917 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:09.779925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:09.780398 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:09.780401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:09.780556 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:09.781008 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:09.781029 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:09.782589 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:09.784500 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:09.784509 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:09.784523 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:09.784755 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:09.785037 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:09.785040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:09.785042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:09.785162 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:09.785603 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:09.785612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:09.785809 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 14:26:09.785971 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:26:09.786234 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":494,"threads_started":1}
I20260620 14:26:09.788048 6692 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 14:26:09.788280 6692 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:09.788771 6692 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 14:26:09.790304 6501 ts_manager.cc:194] Registered new tserver with Master: 0452603959bd46158c77c4f7b2f2ddef (127.0.98.1:45309)
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:26:09.791555 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:26:09.805330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:09.903220 6697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:09.903402 6697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:09.903456 6697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:09.907142 6697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:09.907199 6697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:09.907305 6697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:09.912086 6697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:09.912860 6697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:09.913687 6697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:09.921733 6705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:09.921775 6703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:09.921779 6702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:09.921901 6697 server_base.cc:1034] running on GCE node
I20260620 14:26:09.922253 6697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:09.922791 6697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:09.923972 6697 hybrid_clock.cc:648] HybridClock initialized: now 1781965569923954 us; error 29 us; skew 500 ppm
Jun 20 14:26:09 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:09.926465 6697 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:26:09.927137 6697 webserver.cc:460] Webserver started at http://127.0.98.2:41569/ using document root <none> and password file <none>
I20260620 14:26:09.927592 6697 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:09.927644 6697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:09.927779 6697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:09.929494 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "62879d5b5a8bd8db56595a438a06ed5f"
server_key_iv: "7589fd71d3cb00cae66d1e13c727f7b6"
server_key_version: "encryptionkey@0"
I20260620 14:26:09.929999 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "62879d5b5a8bd8db56595a438a06ed5f"
server_key_iv: "7589fd71d3cb00cae66d1e13c727f7b6"
server_key_version: "encryptionkey@0"
I20260620 14:26:09.933173 6697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:26:09.935295 6712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:09.936373 6697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:09.936496 6697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "62879d5b5a8bd8db56595a438a06ed5f"
server_key_iv: "7589fd71d3cb00cae66d1e13c727f7b6"
server_key_version: "encryptionkey@0"
I20260620 14:26:09.936595 6697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:09.950515 6697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:09.952942 6697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:09.953122 6697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:09.953661 6697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:09.954397 6697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:09.954465 6697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:09.954520 6697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:09.954571 6697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:09.962553 6697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43435
I20260620 14:26:09.962575 6825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43435 every 8 connection(s)
I20260620 14:26:09.963275 6697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:09.970263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 6697
I20260620 14:26:09.970371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:26:09.970705 392 external_mini_cluster.cc:1428] Setting key 48adb77170a1f2f17c737069a02cc775
Jun 20 14:26:09 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:09.975695 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:09.964785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45597 (local address 127.0.98.62:42021)
0620 14:26:09.964904 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:09.964907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:09.965611 (+ 704us) server_negotiation.cc:407] Connection header received
0620 14:26:09.966857 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:09.966861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:09.966918 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:09.966993 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:09.967998 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:09.968520 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:09.969633 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:09.969861 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:09.969968 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:09.972183 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:09.972186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:09.972189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:09.972213 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:09.973765 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:09.974325 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:09.974327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:09.974329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:09.974393 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:09.974757 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:09.974759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:09.974760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:09.974901 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:09.974970 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:09.975505 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 14:26:09.975587 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:26:09.976774 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:09.965079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:45597)
0620 14:26:09.965479 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:09.965515 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:09.966650 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:09.967137 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:09.967143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:09.967453 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:09.967855 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:09.967866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:09.968650 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:09.968655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:09.969505 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:09.969514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:09.969996 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:09.970000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:09.970162 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:09.970676 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:09.970690 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:09.972028 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:09.973869 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:09.973876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:09.973890 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:09.974231 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:09.974480 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:09.974486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:09.974488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:09.974665 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:09.975000 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:09.975009 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:09.975283 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 14:26:09.975474 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:26:09.975779 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":146,"threads_started":1}
I20260620 14:26:09.977931 6826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 14:26:09.978139 6826 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:09.978528 6826 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 14:26:09.979434 6501 ts_manager.cc:194] Registered new tserver with Master: 3f6e991dbbfa482dafbd9c4d9f6809f0 (127.0.98.2:43435)
I20260620 14:26:09.979941 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45597
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:26:09.993818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:10.093438 6831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:10.093633 6831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:10.093698 6831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:10.097476 6831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:10.097533 6831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:10.097630 6831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:10.102264 6831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:10.103044 6831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:10.103883 6831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:10.112195 6836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:10.112226 6837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:10.112371 6831 server_base.cc:1034] running on GCE node
W20260620 14:26:10.112248 6839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:10.112728 6831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:10.113327 6831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:10.114495 6831 hybrid_clock.cc:648] HybridClock initialized: now 1781965570114481 us; error 30 us; skew 500 ppm
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:10.117019 6831 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:26:10.117827 6831 webserver.cc:460] Webserver started at http://127.0.98.3:43231/ using document root <none> and password file <none>
I20260620 14:26:10.118321 6831 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:10.118405 6831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:10.118592 6831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:10.120308 6831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "5f3847a6bab749679b2632fda04b6f3c"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "adf0782a1ca9c0b8f4b4c423d83f8fe6"
server_key_iv: "ee2211398cd19f59b1dac5799dd7561e"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.120807 6831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5f3847a6bab749679b2632fda04b6f3c"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "adf0782a1ca9c0b8f4b4c423d83f8fe6"
server_key_iv: "ee2211398cd19f59b1dac5799dd7561e"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.123919 6831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:10.125864 6846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:10.126935 6831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:26:10.127063 6831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "5f3847a6bab749679b2632fda04b6f3c"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "adf0782a1ca9c0b8f4b4c423d83f8fe6"
server_key_iv: "ee2211398cd19f59b1dac5799dd7561e"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.127161 6831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:10.153666 6831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:10.156523 6831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:10.156708 6831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:10.157227 6831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:10.157953 6831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:10.158022 6831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:10.158087 6831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:10.158129 6831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:10.165736 6831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37545
I20260620 14:26:10.165760 6959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37545 every 8 connection(s)
I20260620 14:26:10.166472 6831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:10.168243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 6831
I20260620 14:26:10.168360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:26:10.168630 392 external_mini_cluster.cc:1428] Setting key 87da52003683ea92de9eee09f215a5cc
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:10.178267 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.168096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40811 (local address 127.0.98.62:42021)
0620 14:26:10.168256 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.168260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.168973 (+ 713us) server_negotiation.cc:407] Connection header received
0620 14:26:10.170195 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.170198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.170246 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.170330 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.171330 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.171835 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.172875 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.173082 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.173169 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.175181 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.175184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.175186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.175206 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.176379 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.176797 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.176799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.176800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.176867 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.177150 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.177153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.177154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.177617 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.177695 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.178080 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.178163 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:26:10.178884 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.168385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:40811)
0620 14:26:10.168826 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.168873 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.170024 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.170510 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.170518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.170831 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.171216 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.171227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.171937 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.171939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.172774 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.172783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.173243 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.173248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.173388 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.173809 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.173823 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.175051 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.176474 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.176478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.176488 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.176714 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.176945 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.176947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.176948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.177060 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.177704 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.177708 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.177888 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.178025 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.178218 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":125,"threads_started":1}
I20260620 14:26:10.179771 6960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 14:26:10.179973 6960 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:10.180371 6960 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 14:26:10.181331 6499 ts_manager.cc:194] Registered new tserver with Master: 5f3847a6bab749679b2632fda04b6f3c (127.0.98.3:37545)
I20260620 14:26:10.181931 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40811
I20260620 14:26:10.182150 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 14:26:10.204844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40593
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42021,127.0.98.61:40593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:10.305976 6965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:10.306217 6965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:10.306285 6965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:10.310257 6965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:10.310324 6965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:10.310357 6965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:10.310410 6965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:10.310443 6965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:10.315155 6965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42021,127.0.98.61:40593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
W20260620 14:26:10.315352 6965 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:42021,127.0.98.61:40593'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:26:10.316040 6965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:10.316866 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:10.324735 6971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:10.324726 6970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:10.324728 6973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:10.325026 6965 server_base.cc:1034] running on GCE node
I20260620 14:26:10.325562 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:10.326470 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:10.327663 6965 hybrid_clock.cc:648] HybridClock initialized: now 1781965570327653 us; error 34 us; skew 500 ppm
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:10.330264 6965 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:26:10.331071 6965 webserver.cc:460] Webserver started at http://127.0.98.61:42857/ using document root <none> and password file <none>
I20260620 14:26:10.331578 6965 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:10.331707 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:10.331888 6965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:10.333688 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data/instance:
uuid: "b3f1103c8e73480b8b6ffd0cc343f76a"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "1079b9a8ebcfee413ac14ba05c3a6d0c"
server_key_iv: "7ce8e8cb9e266e98f7c9fdeed40077f8"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.334213 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal/instance:
uuid: "b3f1103c8e73480b8b6ffd0cc343f76a"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "1079b9a8ebcfee413ac14ba05c3a6d0c"
server_key_iv: "7ce8e8cb9e266e98f7c9fdeed40077f8"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.337651 6965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:26:10.340003 6980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:10.341073 6965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:26:10.341240 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal
uuid: "b3f1103c8e73480b8b6ffd0cc343f76a"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "1079b9a8ebcfee413ac14ba05c3a6d0c"
server_key_iv: "7ce8e8cb9e266e98f7c9fdeed40077f8"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.341373 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:10.351951 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:10.354761 6965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:10.354945 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:10.360975 6965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40593
I20260620 14:26:10.361270 7032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40593 every 8 connection(s)
I20260620 14:26:10.361833 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:26:10.364202 7033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:10.367437 7033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:26:10.369448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 6965
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:26:10.379900 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.368994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.0.98.62:42021)
0620 14:26:10.369128 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.369133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.370012 (+ 879us) server_negotiation.cc:407] Connection header received
0620 14:26:10.371046 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.371049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.371100 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.371200 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.372041 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.372529 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.373646 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.373869 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.373944 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.376266 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.376280 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.376285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.376316 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.377847 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.378355 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.378360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.378362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.378462 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.378743 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.378748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.378751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.379040 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.379130 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.379622 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.379735 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:26:10.380812 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.369411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54404)
0620 14:26:10.369861 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.369908 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.370868 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.371259 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.371267 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.371659 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.371904 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.371914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.372656 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.372659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.373527 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.373535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.373993 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.373997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.374149 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.374636 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.374649 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.376110 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.377948 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.377957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.377965 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.378238 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.378544 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.378546 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.378547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.378655 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.379137 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.379143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.379386 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.379579 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.379857 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":1,"spinlock_wait_cycles":2944,"thread_start_us":126,"threads_started":1}
I20260620 14:26:10.380812 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.370247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.0.98.61:40593)
0620 14:26:10.370576 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.370584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.370613 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:26:10.370878 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.370894 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.371058 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.371223 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.371905 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.372841 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.373681 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.374069 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.374162 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.376695 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.376704 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.376718 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.376746 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.378370 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.378782 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.378785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.378786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.378838 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.379111 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.379116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.379119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.379383 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.379596 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.379805 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.379914 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":262,"spinlock_wait_cycles":896,"thread_start_us":107,"threads_started":1}
I20260620 14:26:10.382233 7033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:26:10.388695 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.382571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40593 (local address 127.0.0.1:59800)
0620 14:26:10.382703 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.382716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.382792 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.382995 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.382997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.383105 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.383181 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.383185 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.383745 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.383747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.384178 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.384184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.384566 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.384568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.384631 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.384959 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.384978 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.386051 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.387645 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.387647 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.387648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.387812 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.388028 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.388030 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.388031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.388092 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.388350 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.388352 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.388411 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.388479 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.388572 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:26:10.388708 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.382637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.0.98.61:40593)
0620 14:26:10.382716 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.382719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.382755 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:26:10.382876 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.382880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.382909 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.382966 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.383267 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.383668 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.384261 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.384454 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.384545 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.386159 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.386161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.386163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.386180 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.387550 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.387899 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.387901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.387902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.387960 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.388158 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.388159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.388160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.388282 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.388345 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.388525 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.388596 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:26:10.391242 7033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Bootstrap starting.
I20260620 14:26:10.393103 7033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:10.393793 7033 log.cc:824] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:10.395262 7033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: No bootstrap required, opened a new log
I20260620 14:26:10.397883 7033 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } }
I20260620 14:26:10.398046 7033 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:10.398080 7033 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f1103c8e73480b8b6ffd0cc343f76a, State: Initialized, Role: FOLLOWER
I20260620 14:26:10.398391 7033 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } }
I20260620 14:26:10.398721 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } } }
I20260620 14:26:10.398820 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:26:10.399287 7033 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:26:10.403223 7050 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:26:10.403306 7050 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:26:10.404930 6965 master_runner.cc:428] Detected that this master b3f1103c8e73480b8b6ffd0cc343f76a is joining an existing cluster
I20260620 14:26:10.404999 6965 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:40593
I20260620 14:26:10.415153 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.407274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.0.98.62:42021)
0620 14:26:10.407408 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.407415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.407898 (+ 483us) server_negotiation.cc:407] Connection header received
0620 14:26:10.408076 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.408081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.408136 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.408209 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.408724 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.409454 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.410206 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.410449 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.410537 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.411847 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.411851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.411853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.411874 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.413497 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.413949 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.413953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.413955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.414034 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.414317 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.414320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.414321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.414473 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.414561 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.414911 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.415020 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:26:10.415495 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.407318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40593 (local address 127.0.0.1:59810)
0620 14:26:10.407776 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.407787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.407953 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.408227 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.408230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.408455 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.408546 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.408550 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.409490 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.409494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.410362 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.410372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.410988 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.410994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.411131 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.411617 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.411631 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.412008 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.413816 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.413819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.413821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.414050 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.414325 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.414329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.414332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.414437 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.414842 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.414845 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.414922 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.415020 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.415140 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":90,"threads_started":1}
I20260620 14:26:10.415455 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.407377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.0.98.61:40593)
0620 14:26:10.407466 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.407470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.407871 (+ 401us) server_negotiation.cc:407] Connection header received
0620 14:26:10.408025 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.408028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.408062 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.408123 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.408667 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.409385 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.410475 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.410857 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.410961 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.412166 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.412184 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.412187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.412215 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.413709 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.414168 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.414171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.414172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.414230 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.414550 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.414553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.414555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.414751 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.414841 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.415077 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.415297 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":34}
I20260620 14:26:10.415665 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.407542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54412)
0620 14:26:10.407819 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.407831 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.407966 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.408241 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.408244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.408473 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.408588 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.408595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.409531 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.409537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.410088 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.410095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.410652 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.410656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.410832 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.411267 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.411301 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.411708 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.413604 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.413609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.413613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.413849 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.414119 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.414122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.414124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.414208 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.414645 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.414647 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.414749 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.415284 (+ 535us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.415395 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":59,"spinlock_wait_cycles":2816,"thread_start_us":92,"threads_started":1}
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:26:10.427825 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.419455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40593 (local address 127.0.0.1:59822)
0620 14:26:10.420150 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.420194 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.420328 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.420581 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.420587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.420977 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.421077 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.421084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.421711 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.421714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.422569 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.422578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.422927 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.422930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.423028 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.423434 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.423447 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.424786 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.426500 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.426507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.426511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.426713 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.426974 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.426976 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.426977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.427044 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.427403 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.427407 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.427503 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.427605 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.427722 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":80,"threads_started":1}
I20260620 14:26:10.427899 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.419490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.0.98.61:40593)
0620 14:26:10.419565 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.419568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.420281 (+ 713us) server_negotiation.cc:407] Connection header received
0620 14:26:10.420434 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.420436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.420471 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.420532 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.421176 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.421618 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.422674 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.422839 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.422898 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.424906 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.424910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.424912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.424935 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.426400 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.426806 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.426810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.426812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.426889 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.427123 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.427126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.427127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.427306 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.427400 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.427673 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.427766 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:26:10.429036 6499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:40593: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 40593
}
attrs {
promote: true
}
}
dest_uuid: "9ef79ad2a6de42ecacfb6489b5381f63"
cas_config_opid_index: -1
I20260620 14:26:10.429638 6499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } }
I20260620 14:26:10.431048 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b3f1103c8e73480b8b6ffd0cc343f76a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } } }
I20260620 14:26:10.431732 7007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:10.432679 7007 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Refusing update from remote peer 9ef79ad2a6de42ecacfb6489b5381f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:26:10.432700 7059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:40593
I20260620 14:26:10.432889 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } } }
I20260620 14:26:10.432974 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:10.433082 6965 master.cc:537] Master@127.0.98.61:40593 shutting down...
I20260620 14:26:10.433470 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:26:10.435043 6488 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 14:26:10.435829 6488 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 -> Peer b3f1103c8e73480b8b6ffd0cc343f76a (127.0.98.61:40593): Couldn't send request to peer b3f1103c8e73480b8b6ffd0cc343f76a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:26:10.437698 6965 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:10.437953 6965 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:10.438057 6965 tablet_replica.cc:330] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: stopping tablet replica
W20260620 14:26:10.440657 6984 connection.cc:441] server connection from 127.0.0.1:59798 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:59798 (request call id 15) could send its response
I20260620 14:26:10.452214 6965 master.cc:552] Master@127.0.98.61:40593 shutdown complete.
I20260620 14:26:10.452251 6965 master_runner.cc:305] Clearing existing system tablet
I20260620 14:26:10.452453 6965 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:10.452494 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:10.453506 6965 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:26:10.454291 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:10.454524 6965 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:10.454645 6965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:10.454690 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal
uuid: "b3f1103c8e73480b8b6ffd0cc343f76a"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "1079b9a8ebcfee413ac14ba05c3a6d0c"
server_key_iv: "7ce8e8cb9e266e98f7c9fdeed40077f8"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.456094 6965 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:10.457849 6965 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:26:10.457945 6965 log.cc:1196] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:26:10.458174 6965 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Deleting consensus metadata
I20260620 14:26:10.458338 6965 master_runner.cc:315] Copying system tablet from 127.0.98.62:42021
I20260620 14:26:10.459318 6965 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42021
I20260620 14:26:10.466722 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.459821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.0.98.62:42021)
0620 14:26:10.459943 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.459949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.460107 (+ 158us) server_negotiation.cc:407] Connection header received
0620 14:26:10.460203 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.460205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.460238 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.460299 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.461073 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.461614 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.462658 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.462835 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.462901 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.463957 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.463959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.463961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.463980 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.465384 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.465781 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.465783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.465784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.465846 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.466074 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.466076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.466077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.466211 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.466277 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.466548 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.466624 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:26:10.466820 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.459760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54424)
0620 14:26:10.460013 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.460027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.460126 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.460334 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.460337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.460545 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.460967 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.460975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.461704 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.461707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.462557 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.462566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.462924 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.462927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.463021 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.463451 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.463464 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.463835 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.465473 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.465476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.465478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.465692 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.465917 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.465920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.465921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.465999 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.466288 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.466291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.466409 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.466510 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.466633 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":114,"threads_started":1}
I20260620 14:26:10.467458 6521 tablet_copy_service.cc:140] P 9ef79ad2a6de42ecacfb6489b5381f63: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b3f1103c8e73480b8b6ffd0cc343f76a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54424)
I20260620 14:26:10.467628 6521 tablet_copy_service.cc:161] P 9ef79ad2a6de42ecacfb6489b5381f63: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b3f1103c8e73480b8b6ffd0cc343f76a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54424: session id = b3f1103c8e73480b8b6ffd0cc343f76a-00000000000000000000000000000000
I20260620 14:26:10.468788 6521 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:26:10.470073 6965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:26:10.471484 6965 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: tablet copy: Starting download of 0 data blocks...
I20260620 14:26:10.471678 6965 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: tablet copy: Starting download of 1 WAL segments...
I20260620 14:26:10.473024 6965 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:26:10.474337 6521 tablet_copy_service.cc:342] P 9ef79ad2a6de42ecacfb6489b5381f63: Request end of tablet copy session b3f1103c8e73480b8b6ffd0cc343f76a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54424
I20260620 14:26:10.474414 6521 tablet_copy_service.cc:433] P 9ef79ad2a6de42ecacfb6489b5381f63: ending tablet copy session b3f1103c8e73480b8b6ffd0cc343f76a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b3f1103c8e73480b8b6ffd0cc343f76a
I20260620 14:26:10.479992 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:10.481292 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:10.481426 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:10.481534 6965 server_base.cc:1034] running on GCE node
W20260620 14:26:10.481626 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:10.481838 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:10.482096 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:10.483232 6965 hybrid_clock.cc:648] HybridClock initialized: now 1781965570483217 us; error 28 us; skew 500 ppm
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:10.484432 6965 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 14:26:10.484853 6965 webserver.cc:460] Webserver started at http://127.0.98.61:37013/ using document root <none> and password file <none>
I20260620 14:26:10.485066 6965 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:10.485128 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:10.486069 6965 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:10.486913 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:10.487124 6965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:10.487188 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal
uuid: "b3f1103c8e73480b8b6ffd0cc343f76a"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "1079b9a8ebcfee413ac14ba05c3a6d0c"
server_key_iv: "7ce8e8cb9e266e98f7c9fdeed40077f8"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.487241 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:10.507951 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:10.509752 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:10.514380 6965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40593
I20260620 14:26:10.514400 7133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40593 every 8 connection(s)
I20260620 14:26:10.514848 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:26:10.516059 7134 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:26:10.516829 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Bootstrap starting.
I20260620 14:26:10.524992 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:26:10.525296 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Bootstrap complete.
I20260620 14:26:10.525738 7134 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } }
I20260620 14:26:10.525858 7134 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3f1103c8e73480b8b6ffd0cc343f76a, State: Initialized, Role: LEARNER
I20260620 14:26:10.525944 7134 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } }
I20260620 14:26:10.526212 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } } }
I20260620 14:26:10.526324 7134 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:10.526419 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:26:10.529479 7148 catalog_manager.cc:1259] Loaded cluster ID: 6c8df4cacf1b4e0d8cc811934d63ba7c
I20260620 14:26:10.529533 7148 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: loading cluster ID for follower catalog manager: success
I20260620 14:26:10.531491 7148 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: acquiring CA information for follower catalog manager: success
I20260620 14:26:10.532879 7148 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:26:10.549744 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.542216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.0.98.61:40593)
0620 14:26:10.542435 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.542439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.542454 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:26:10.542552 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.542554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.542592 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.542681 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.543346 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.544313 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.545194 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.545523 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.545629 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.546861 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.546865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.546867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.546890 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.548352 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.548685 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.548688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.548690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.548763 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.548976 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.548979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.548980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.549141 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.549282 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.549467 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.549566 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 14:26:10.572153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41443
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42021,127.0.98.61:40593,127.0.98.60:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:10.672552 7152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:10.672792 7152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:10.672857 7152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:10.676561 7152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:10.676638 7152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:10.676676 7152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:10.676739 7152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:10.676779 7152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:10.681502 7152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42021,127.0.98.61:40593,127.0.98.60:41443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:10.682315 7152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:10.683207 7152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:10.691268 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:10.691291 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:10.691408 7152 server_base.cc:1034] running on GCE node
W20260620 14:26:10.691562 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:10.692081 7152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:10.692965 7152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:10.694114 7152 hybrid_clock.cc:648] HybridClock initialized: now 1781965570694086 us; error 40 us; skew 500 ppm
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:10.696434 7152 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:26:10.697335 7152 webserver.cc:460] Webserver started at http://127.0.98.60:45997/ using document root <none> and password file <none>
I20260620 14:26:10.697837 7152 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:10.697921 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:10.698112 7152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:10.699843 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data/instance:
uuid: "a7ebe6ca063b4c5b8234e63518c59383"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "342f2155252c9504f6e20fadeb7c196d"
server_key_iv: "acc40f663eda55a2867a0003788587c2"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.700352 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal/instance:
uuid: "a7ebe6ca063b4c5b8234e63518c59383"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "342f2155252c9504f6e20fadeb7c196d"
server_key_iv: "acc40f663eda55a2867a0003788587c2"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.703631 7152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 14:26:10.705703 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:10.706811 7152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 14:26:10.706931 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal
uuid: "a7ebe6ca063b4c5b8234e63518c59383"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "342f2155252c9504f6e20fadeb7c196d"
server_key_iv: "acc40f663eda55a2867a0003788587c2"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.707031 7152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:10.750321 7152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:10.752761 7152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:10.752944 7152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:10.759400 7152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41443
I20260620 14:26:10.759645 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41443 every 8 connection(s)
I20260620 14:26:10.760262 7152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:26:10.762698 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:10.765827 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:26:10.767608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 7152
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:26:10.779461 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.767465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.0.98.62:42021)
0620 14:26:10.767639 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.767645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.768243 (+ 598us) server_negotiation.cc:407] Connection header received
0620 14:26:10.769293 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.769296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.769336 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.769398 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.770190 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.770730 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.772319 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.772539 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.772613 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.775751 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.775754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.775756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.775779 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.777508 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.778036 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.778039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.778040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.778142 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.778485 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.778488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.778490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.778727 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.778821 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.779190 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.779307 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:26:10.780153 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.767737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54426)
0620 14:26:10.768116 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.768159 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.769073 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.769467 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.769472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.769842 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.770058 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.770069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.771369 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.771373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.772191 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.772200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.772664 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.772669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.772820 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.773661 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.773675 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.775589 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.777615 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.777621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.777629 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.777929 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.778227 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.778229 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.778230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.778392 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.778868 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.778874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.778990 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.779146 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.779403 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":58,"thread_start_us":112,"threads_started":1}
I20260620 14:26:10.780153 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.768306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.0.98.60:41443)
0620 14:26:10.768564 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.768577 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.768627 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:26:10.769073 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.769093 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.769328 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.769503 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.770227 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.771551 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.772352 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.772748 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.772902 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.775701 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.775715 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.775726 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.775748 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.777642 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.778040 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.778045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.778047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.778192 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.778485 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.778488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.778490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.778906 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.779079 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.779929 (+ 850us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.780024 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
I20260620 14:26:10.781234 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:26:10.788199 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.781602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40593 (local address 127.0.0.1:59846)
0620 14:26:10.781695 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.781708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.781770 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.782021 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.782023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.782166 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.782235 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.782240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.782970 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.782973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.783436 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.783461 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.783896 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.783899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.783983 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.784339 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.784348 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.785374 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.787058 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.787060 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.787061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.787226 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.787495 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.787497 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.787498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.787563 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.787880 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.787882 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.787960 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.788031 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.788111 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:26:10.788304 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.781715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.0.98.61:40593)
0620 14:26:10.781828 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.781832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.781845 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:10.781886 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.781888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.781932 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.782010 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.782375 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.782865 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.783578 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.783798 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.783897 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.785482 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.785485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.785486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.785506 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.786973 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.787303 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.787306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.787307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.787411 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.787644 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.787646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.787647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.787781 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.787878 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.788091 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.788176 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:26:10.788887 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:26:10.793582 6692 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
I20260620 14:26:10.795516 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.789273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41443 (local address 127.0.0.1:42872)
0620 14:26:10.789343 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.789353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.789427 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.789597 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.789600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.789722 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.789772 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.789776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.790378 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.790383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.790800 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.790806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.791319 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.791322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.791433 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.791788 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.791798 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.792770 (+ 972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.794318 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.794323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.794325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.794486 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.794718 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.794723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.794724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.794794 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.795095 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.795098 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.795164 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.795269 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.795376 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":16}
I20260620 14:26:10.795529 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.789362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.0.98.60:41443)
0620 14:26:10.789444 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.789450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.789462 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:26:10.789491 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.789493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.789520 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.789582 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.789867 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.790287 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.790931 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.791215 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.791322 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.792888 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.792891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.792893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.792913 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.794221 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.794567 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.794573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.794576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.794638 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.794872 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.794876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.794878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.795009 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.795102 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.795309 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.795421 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":34}
I20260620 14:26:10.798206 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: Bootstrap starting.
I20260620 14:26:10.800173 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:10.800798 7220 log.cc:824] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:10.802316 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: No bootstrap required, opened a new log
I20260620 14:26:10.804832 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } }
I20260620 14:26:10.804996 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:10.805025 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ebe6ca063b4c5b8234e63518c59383, State: Initialized, Role: FOLLOWER
I20260620 14:26:10.805425 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } }
I20260620 14:26:10.805801 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } } }
I20260620 14:26:10.805913 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:26:10.806615 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:26:10.810706 7236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:26:10.810783 7236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:26:10.813426 7152 master_runner.cc:428] Detected that this master a7ebe6ca063b4c5b8234e63518c59383 is joining an existing cluster
I20260620 14:26:10.813494 7152 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:41443
I20260620 14:26:10.832926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 6563
I20260620 14:26:10.840343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45309
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42021
--webserver_port=34783
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42021,127.0.98.61:40593
--tserver_master_addrs=127.0.98.62:42021,127.0.98.61:40593,127.0.98.60:41443 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:26:10.843423 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.815866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.0.98.62:42021)
0620 14:26:10.815991 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.815997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.816010 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:10.816039 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.816041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.816075 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.816138 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.816878 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.817669 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.818963 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.819273 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.819347 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.839917 (+ 20570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.839921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.839925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.839952 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.841557 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.842074 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.842077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.842079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.842155 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.842467 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.842471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.842476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.842698 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.842764 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.843132 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.843250 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:26:10.843931 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.815125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54436)
0620 14:26:10.815474 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.815487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.815607 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.816238 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.816242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.816552 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.816749 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.816755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.817806 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.817809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.818827 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.818837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.819499 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.819502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.819729 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.837728 (+ 17999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.837744 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.839776 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.841673 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.841676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.841677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.841970 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.842275 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.842279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.842283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.842367 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.842834 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.842839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.842941 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.843541 (+ 600us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.843693 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":214,"spinlock_wait_cycles":1408,"thread_start_us":80,"threads_started":1}
I20260620 14:26:10.844411 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.815835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.0.98.60:41443)
0620 14:26:10.815925 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.815929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.816130 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:26:10.816276 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.816279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.816316 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.816359 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.816929 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.817698 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.819008 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.819322 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.819389 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.837890 (+ 18501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.837894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.837896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.837918 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.839648 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.841902 (+ 2254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.841908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.841910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.841970 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.842848 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.842851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.842852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.843019 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.843080 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.844211 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.844304 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:26:10.844949 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.815506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41443 (local address 127.0.0.1:42876)
0620 14:26:10.816060 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.816072 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.816179 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.816463 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.816466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.816705 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.816813 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.816820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.817826 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.817829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.818893 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.818903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.819511 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.819514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.819649 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.820054 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.820072 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.837776 (+ 17704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.840250 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.840255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.840260 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.841771 (+ 1511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.842291 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.842293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.842294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.842568 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.843134 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.843138 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.843208 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.844690 (+ 1482us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.844813 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":154,"thread_start_us":124,"threads_started":1}
I20260620 14:26:10.845629 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.815212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.0.98.61:40593)
0620 14:26:10.815311 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.815315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.815871 (+ 556us) server_negotiation.cc:407] Connection header received
0620 14:26:10.816030 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.816033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.816074 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.816124 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.816878 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.817677 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.818980 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.819299 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.819359 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.839077 (+ 19718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.839083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.839087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.839115 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.841557 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.842359 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.842362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.842364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.842439 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.843337 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.843340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.843342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.843532 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.843598 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.845395 (+ 1797us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.845492 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:26:10.846699 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.815168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40593 (local address 127.0.0.1:59858)
0620 14:26:10.815796 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.815809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.815927 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.816274 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.816279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.816584 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.816748 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.816755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.817806 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.817809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.818866 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.818876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.819499 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.819502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.819708 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.820445 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.820456 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.838620 (+ 18164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.841676 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.841678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.841680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.842189 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.842664 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.842666 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.842668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.842755 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.843936 (+ 1181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.843942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.845221 (+ 1279us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.846417 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.846543 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":447,"spinlock_wait_cycles":2304,"thread_start_us":86,"threads_started":1}
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965569, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:26:10.854358 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.847347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41443 (local address 127.0.0.1:42884)
0620 14:26:10.847556 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.847573 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.847645 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.848050 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.848053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.848217 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.848282 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.848287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.848897 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.848902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.849427 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.849434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.849835 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.849838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.849921 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.850312 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.850322 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.851387 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.852951 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.852954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.852956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.853136 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.853487 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.853490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.853492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.853573 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.853903 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.853906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.853994 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.854093 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.854215 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260620 14:26:10.854379 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.847757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.0.98.60:41443)
0620 14:26:10.847860 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.847869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.847878 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:26:10.847918 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.847919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.847951 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.848037 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.848382 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.848801 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.849529 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.849736 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.849834 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.851509 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.851512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.851513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.851530 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.852835 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.853315 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.853321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.853324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.853393 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.853665 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.853668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.853669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.853811 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.853908 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.854147 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.854266 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:26:10.855300 6499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:41443: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 41443
}
attrs {
promote: true
}
}
dest_uuid: "9ef79ad2a6de42ecacfb6489b5381f63"
cas_config_opid_index: 5
I20260620 14:26:10.855792 6499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } }
I20260620 14:26:10.856734 7060 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.857766 7194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:10.858249 7060 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:41443
I20260620 14:26:10.858389 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.858467 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:10.858798 7152 master.cc:537] Master@127.0.98.60:41443 shutting down...
I20260620 14:26:10.859098 7194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 FOLLOWER]: Refusing update from remote peer 9ef79ad2a6de42ecacfb6489b5381f63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:26:10.859594 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:26:10.861385 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 -> Peer a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60:41443): Couldn't send request to peer a7ebe6ca063b4c5b8234e63518c59383. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:26:10.863449 7152 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:10.863715 7152 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:10.863826 7152 tablet_replica.cc:330] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: stopping tablet replica
I20260620 14:26:10.866043 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.856883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40593 (local address 127.0.0.1:59874)
0620 14:26:10.861609 (+ 4726us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.861622 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.861695 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.862199 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.862202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.862220 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:10.862288 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.862293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.863210 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.863213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.864620 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.864629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.865660 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.865663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.865753 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.865793 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.865862 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.865920 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4668}
I20260620 14:26:10.866503 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.857070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.0.98.61:40593)
0620 14:26:10.861534 (+ 4464us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.861538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.861791 (+ 253us) server_negotiation.cc:407] Connection header received
0620 14:26:10.861834 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.861836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.861891 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.861988 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:10.862398 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.863088 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.864747 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.865319 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.865410 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.865562 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.866261 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.866365 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4396}
I20260620 14:26:10.867475 7100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 LEARNER]: Refusing update from remote peer 9ef79ad2a6de42ecacfb6489b5381f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:26:10.867878 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 14:26:10.869088 7152 master.cc:552] Master@127.0.98.60:41443 shutdown complete.
I20260620 14:26:10.869122 7152 master_runner.cc:305] Clearing existing system tablet
I20260620 14:26:10.869370 7152 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:10.869415 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:10.869699 7100 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.871541 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef79ad2a6de42ecacfb6489b5381f63. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.871645 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:26:10.871734 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: true } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.871805 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: This master's current role is: LEARNER
I20260620 14:26:10.871917 7059 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63: attempting to promote NON_VOTER b3f1103c8e73480b8b6ffd0cc343f76a to VOTER
I20260620 14:26:10.872330 7152 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.000s
I20260620 14:26:10.872319 7059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } }
I20260620 14:26:10.873618 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.873404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41443 (local address 127.0.0.1:42886)
0620 14:26:10.873527 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.873573 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 14:26:10.874092 7250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:10.874251 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.874035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41443 (local address 127.0.0.1:42900)
0620 14:26:10.874150 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.874194 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 14:26:10.874267 7152 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:10.874377 7152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260620 14:26:10.874408 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal
uuid: "a7ebe6ca063b4c5b8234e63518c59383"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "342f2155252c9504f6e20fadeb7c196d"
server_key_iv: "acc40f663eda55a2867a0003788587c2"
server_key_version: "encryptionkey@0"
W20260620 14:26:10.874536 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 -> Peer a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60:41443): Couldn't send request to peer a7ebe6ca063b4c5b8234e63518c59383. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41443: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:26:10.874893 7100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 LEARNER]: Refusing update from remote peer 9ef79ad2a6de42ecacfb6489b5381f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:26:10.875254 7249 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 14:26:10.876188 7152 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:10.876716 7060 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b3f1103c8e73480b8b6ffd0cc343f76a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.878139 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.878232 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:10.878502 7152 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:26:10.878579 7152 log.cc:1196] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:26:10.878820 7152 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: Deleting consensus metadata
I20260620 14:26:10.878968 7152 master_runner.cc:315] Copying system tablet from 127.0.98.62:42021
I20260620 14:26:10.878988 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.879078 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:10.879693 7100 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b3f1103c8e73480b8b6ffd0cc343f76a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.880012 7152 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42021
I20260620 14:26:10.880775 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.880882 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:26:10.891391 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.881568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.0.98.62:42021)
0620 14:26:10.881666 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:10.881670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:10.884995 (+ 3325us) server_negotiation.cc:407] Connection header received
0620 14:26:10.885115 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:10.885119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:10.885158 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:10.885255 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:10.885639 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.886207 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.887102 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.887332 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.887445 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.888574 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:10.888578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:10.888580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:10.888600 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:10.889930 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.890334 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.890338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.890340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.890414 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:10.890669 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:10.890674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:10.890676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:10.890827 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:10.890927 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:10.891160 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:26:10.891259 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:26:10.891408 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:10.881489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54440)
0620 14:26:10.884917 (+ 3428us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:10.884928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:10.885028 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:10.885270 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:10.885272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:10.885455 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:10.885532 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.885538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.886308 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.886314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.886924 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:10.886933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:10.887463 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:10.887467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:10.887564 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:10.888047 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:10.888065 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:10.888415 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:10.890051 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.890055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.890057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.890242 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.890499 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:10.890503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:10.890505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:10.890586 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:10.890918 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:10.890921 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:10.891007 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 14:26:10.891113 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:26:10.891234 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3360,"thread_start_us":52,"threads_started":1}
I20260620 14:26:10.892000 6521 tablet_copy_service.cc:140] P 9ef79ad2a6de42ecacfb6489b5381f63: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a7ebe6ca063b4c5b8234e63518c59383 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54440)
I20260620 14:26:10.892102 6521 tablet_copy_service.cc:161] P 9ef79ad2a6de42ecacfb6489b5381f63: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a7ebe6ca063b4c5b8234e63518c59383 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54440: session id = a7ebe6ca063b4c5b8234e63518c59383-00000000000000000000000000000000
I20260620 14:26:10.892802 6521 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:26:10.893692 7152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:26:10.895172 7152 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: tablet copy: Starting download of 0 data blocks...
I20260620 14:26:10.895355 7152 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: tablet copy: Starting download of 1 WAL segments...
I20260620 14:26:10.896564 7152 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:26:10.897919 6521 tablet_copy_service.cc:342] P 9ef79ad2a6de42ecacfb6489b5381f63: Request end of tablet copy session a7ebe6ca063b4c5b8234e63518c59383-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54440
I20260620 14:26:10.898017 6521 tablet_copy_service.cc:433] P 9ef79ad2a6de42ecacfb6489b5381f63: ending tablet copy session a7ebe6ca063b4c5b8234e63518c59383-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a7ebe6ca063b4c5b8234e63518c59383
I20260620 14:26:10.903240 7152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:10.904606 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:10.904783 7152 server_base.cc:1034] running on GCE node
W20260620 14:26:10.904798 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:10.905030 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:10.905243 7152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:10.905503 7152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:10.906647 7152 hybrid_clock.cc:648] HybridClock initialized: now 1781965570906643 us; error 18 us; skew 500 ppm
Jun 20 14:26:10 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965570, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:10.908138 7152 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 14:26:10.908563 7152 webserver.cc:460] Webserver started at http://127.0.98.60:40899/ using document root <none> and password file <none>
I20260620 14:26:10.908732 7152 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:10.908785 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:10.909854 7152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:10.910624 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:10.910833 7152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:10.910912 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal
uuid: "a7ebe6ca063b4c5b8234e63518c59383"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "342f2155252c9504f6e20fadeb7c196d"
server_key_iv: "acc40f663eda55a2867a0003788587c2"
server_key_version: "encryptionkey@0"
I20260620 14:26:10.910976 7152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:10.929001 7152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:10.930929 7152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:10.935439 7152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41443
I20260620 14:26:10.935961 7152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:26:10.937434 7321 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:26:10.938165 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: Bootstrap starting.
I20260620 14:26:10.943838 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41443 every 8 connection(s)
I20260620 14:26:10.946604 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:26:10.946887 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: Bootstrap complete.
I20260620 14:26:10.947335 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } }
I20260620 14:26:10.947476 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7ebe6ca063b4c5b8234e63518c59383, State: Initialized, Role: LEARNER
I20260620 14:26:10.947603 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } }
I20260620 14:26:10.948087 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:10.950013 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:10.950138 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:26:10.950920 7335 catalog_manager.cc:1259] Loaded cluster ID: 6c8df4cacf1b4e0d8cc811934d63ba7c
I20260620 14:26:10.951001 7335 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: loading cluster ID for follower catalog manager: success
I20260620 14:26:10.952584 7335 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: acquiring CA information for follower catalog manager: success
I20260620 14:26:10.953629 7335 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:26:10.981675 6826 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
W20260620 14:26:10.987820 7245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:10.988081 7245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:10.988150 7245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:10.994072 7245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:10.994136 7245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:10.994212 7245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:11.001922 7245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34783
--tserver_master_addrs=127.0.98.62:42021,127.0.98.61:40593,127.0.98.60:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:11.002990 7245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:11.004078 7245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:11.013857 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:11.013890 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:11.013868 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:11.014052 7245 server_base.cc:1034] running on GCE node
I20260620 14:26:11.014501 7245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:11.015049 7245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:11.016237 7245 hybrid_clock.cc:648] HybridClock initialized: now 1781965571016215 us; error 35 us; skew 500 ppm
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:11.018821 7245 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:26:11.019507 7245 webserver.cc:460] Webserver started at http://127.0.98.1:34783/ using document root <none> and password file <none>
I20260620 14:26:11.020076 7245 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:11.020148 7245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:11.023566 7245 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:11.025378 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:11.026531 7245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:11.026647 7245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "0452603959bd46158c77c4f7b2f2ddef"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "ea9c39aca30cf799805e7c8c102e5bd5"
server_key_iv: "9eafda8a59759f1b79bcc2ceb4a5248c"
server_key_version: "encryptionkey@0"
I20260620 14:26:11.027166 7245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:11.043701 7245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:11.046777 7245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:11.046906 7245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:11.047348 7245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:11.048121 7245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:11.048166 7245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:11.048200 7245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:11.048220 7245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:11.057085 7245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45309
I20260620 14:26:11.057109 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45309 every 8 connection(s)
I20260620 14:26:11.057972 7245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:26:11.066702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 7245
I20260620 14:26:11.066890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 6697
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:11.072571 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.060935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34579 (local address 127.0.98.61:40593)
0620 14:26:11.061052 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.061056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.061229 (+ 173us) server_negotiation.cc:407] Connection header received
0620 14:26:11.061492 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.061495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.061558 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.061628 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.062820 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.063578 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.064814 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.065130 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.065246 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.068561 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.068568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.068574 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.068605 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.070364 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.070940 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.070944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.070945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.071030 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.071397 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.071400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.071402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.071654 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.071721 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.072309 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.072418 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:26:11.072633 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.059472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53893 (local address 127.0.98.60:41443)
0620 14:26:11.059744 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.059757 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.060452 (+ 695us) server_negotiation.cc:407] Connection header received
0620 14:26:11.061516 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.061519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.061576 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.061632 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.062855 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.063807 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.064660 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.064985 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.065088 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.068873 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.068876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.068879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.068902 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.070970 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.071397 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.071400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.071402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.071493 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.071801 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.071803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.071805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.071989 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.072058 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.072370 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.072475 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":74,"threads_started":1}
I20260620 14:26:11.073455 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.059496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49463 (local address 127.0.98.62:42021)
0620 14:26:11.059609 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.059613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.060816 (+ 1203us) server_negotiation.cc:407] Connection header received
0620 14:26:11.061589 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.061592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.061643 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.061706 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.062844 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.063347 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.064842 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.065119 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.065234 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.069357 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.069364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.069367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.069395 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.071184 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.071724 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.071730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.071735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.071828 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.072125 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.072131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.072135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.072351 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.072441 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.073091 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.073237 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:26:11.074002 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.060847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40593 (local address 127.0.98.1:34579)
0620 14:26:11.061113 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.061125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.061309 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.061776 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.061781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.062180 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:11.062689 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.062701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.063681 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.063684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.064703 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.064713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.065365 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.065368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.065517 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.066395 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:11.066409 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:11.068397 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:11.070493 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.070501 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.070516 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.070822 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.071149 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.071153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.071155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.071293 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.071795 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:11.071800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:11.072058 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.072715 (+ 657us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.073091 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":167,"spinlock_wait_cycles":27264,"thread_start_us":89,"threads_started":1}
I20260620 14:26:11.074234 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.059852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41443 (local address 127.0.98.1:53893)
0620 14:26:11.060301 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.060346 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.061305 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.061796 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.061798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.062178 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:11.062705 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.062711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.063907 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.063912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.064544 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.064552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.065255 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.065259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.065435 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.066165 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:11.066179 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:11.068747 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:11.071079 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.071083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.071085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.071292 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.071616 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.071619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.071621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.071704 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.072127 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:11.072129 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:11.072203 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.072832 (+ 629us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.073143 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":215,"spinlock_wait_cycles":8960,"thread_start_us":491,"threads_started":1}
I20260620 14:26:11.074216 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.059853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:49463)
0620 14:26:11.060733 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.060745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.061305 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.061804 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.061806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.062190 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:11.062703 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.062711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.063478 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.063481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.064703 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.064713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.065348 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.065356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.065441 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.066089 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:11.066103 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:11.069185 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:11.071305 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.071311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.071316 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.071608 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.071929 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.071934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.071939 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.072028 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.072483 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:11.072489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:11.072596 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.072918 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.073821 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":907,"spinlock_wait_cycles":4864,"thread_start_us":129,"threads_started":1}
I20260620 14:26:11.075309 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 14:26:11.075294 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41443
I20260620 14:26:11.075503 7464 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:11.075505 7465 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:11.075284 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40593
I20260620 14:26:11.075850 7466 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:11.075956 7466 heartbeater.cc:511] Master 127.0.98.61:40593 requested a full tablet report, sending...
I20260620 14:26:11.076018 7464 heartbeater.cc:511] Master 127.0.98.60:41443 requested a full tablet report, sending...
I20260620 14:26:11.075959 7465 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 14:26:11.077265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43435
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42021
--webserver_port=41569
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42021,127.0.98.61:40593
--tserver_master_addrs=127.0.98.62:42021,127.0.98.61:40593,127.0.98.60:41443 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:26:11.077337 7285 ts_manager.cc:194] Registered new tserver with Master: 0452603959bd46158c77c4f7b2f2ddef (127.0.98.1:45309)
I20260620 14:26:11.077432 6499 ts_manager.cc:194] Re-registered known tserver with Master: 0452603959bd46158c77c4f7b2f2ddef (127.0.98.1:45309)
I20260620 14:26:11.078106 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49463
I20260620 14:26:11.078243 7092 ts_manager.cc:194] Registered new tserver with Master: 0452603959bd46158c77c4f7b2f2ddef (127.0.98.1:45309)
W20260620 14:26:11.180371 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:11.180590 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:11.180655 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:26:11.183947 6960 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
W20260620 14:26:11.184412 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:11.184451 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:11.184542 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:11.189940 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41569
--tserver_master_addrs=127.0.98.62:42021,127.0.98.61:40593,127.0.98.60:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:11.190788 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:11.191723 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:11.200457 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:11.200412 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:11.200416 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:11.200688 7472 server_base.cc:1034] running on GCE node
I20260620 14:26:11.201085 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:11.201653 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:11.202807 7472 hybrid_clock.cc:648] HybridClock initialized: now 1781965571202778 us; error 39 us; skew 500 ppm
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:11.205281 7472 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:26:11.206097 7472 webserver.cc:460] Webserver started at http://127.0.98.2:41569/ using document root <none> and password file <none>
I20260620 14:26:11.206615 7472 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:11.206704 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:11.210057 7472 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:26:11.211856 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:11.212986 7472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:11.213097 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0"
format_stamp: "Formatted at 2026-06-20 14:26:09 on dist-test-slave-pk3j"
server_key: "62879d5b5a8bd8db56595a438a06ed5f"
server_key_iv: "7589fd71d3cb00cae66d1e13c727f7b6"
server_key_version: "encryptionkey@0"
I20260620 14:26:11.213604 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:11.247126 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:11.250072 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:11.250252 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:11.250813 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:11.251595 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:11.251643 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:11.251680 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:11.251725 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:11.260179 7472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43435
I20260620 14:26:11.260203 7600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43435 every 8 connection(s)
I20260620 14:26:11.261070 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:11.264228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 7472
I20260620 14:26:11.264406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 6831
I20260620 14:26:11.271723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37545
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42021
--webserver_port=43231
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42021,127.0.98.61:40593
--tserver_master_addrs=127.0.98.62:42021,127.0.98.61:40593,127.0.98.60:41443 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 14:26:11.278743 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.262706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60205 (local address 127.0.98.62:42021)
0620 14:26:11.262820 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.262825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.263790 (+ 965us) server_negotiation.cc:407] Connection header received
0620 14:26:11.265649 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.265652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.265705 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.265757 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.267422 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.268211 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.270196 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.270524 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.270591 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.274032 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.274035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.274037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.274061 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.276906 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.277540 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.277544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.277546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.277644 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.277924 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.277930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.277933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.278118 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.278193 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.278509 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.278616 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:26:11.278898 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.263174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41443 (local address 127.0.98.2:48447)
0620 14:26:11.264157 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.264171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.266283 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.266749 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.266755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.266968 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:11.267276 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.267288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.268463 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.268469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.270287 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.270299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.271117 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.271123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.271239 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.272208 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:11.272227 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:11.274347 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:11.275816 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.275821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.275833 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.276136 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.276472 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.276474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.276476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.276641 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.277147 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:11.277153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:11.277471 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.277662 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.277976 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":759,"spinlock_wait_cycles":4992,"thread_start_us":86,"threads_started":1}
I20260620 14:26:11.279101 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.262915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45273 (local address 127.0.98.61:40593)
0620 14:26:11.263071 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.263075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.264151 (+ 1076us) server_negotiation.cc:407] Connection header received
0620 14:26:11.265397 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.265403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.265461 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.265516 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.267421 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.268211 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.270195 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.270520 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.270591 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.273887 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.273890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.273893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.273918 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.276882 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.277610 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.277616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.277619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.277710 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.278045 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.278049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.278051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.278280 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.278373 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.278769 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.278894 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:26:11.279640 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.263339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48447 (local address 127.0.98.60:41443)
0620 14:26:11.263466 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.263470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.264246 (+ 776us) server_negotiation.cc:407] Connection header received
0620 14:26:11.266454 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.266461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.266524 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.266587 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.267419 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.268321 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.270417 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.270848 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.270930 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.274452 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.274455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.274457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.274482 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.275709 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.276251 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.276254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.276257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.276343 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.276771 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.276777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.276780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.277008 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.277095 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.279360 (+ 2265us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.279481 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:26:11.280033 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.263147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:60205)
0620 14:26:11.263646 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.263697 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.265156 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.265895 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.265903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.266480 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:11.267264 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.267275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.268387 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.268390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.270061 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.270071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.270771 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.270774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.271061 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.272074 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:11.272087 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:11.273870 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:11.277018 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.277021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.277022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.277423 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.277741 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.277743 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.277745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.277822 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.278258 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:11.278261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:11.278340 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.279371 (+ 1031us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.279597 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":575,"spinlock_wait_cycles":13824,"thread_start_us":125,"threads_started":1}
I20260620 14:26:11.280196 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.263147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40593 (local address 127.0.98.2:45273)
0620 14:26:11.264072 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.264085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.265156 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.265895 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.265903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.266480 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:11.267255 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.267275 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.268387 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.268390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.270066 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.270071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.270771 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.270774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.271058 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.271834 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:11.271859 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:11.273706 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:11.277021 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.277026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.277029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.277458 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.277833 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.277838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.277841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.277931 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.278436 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:11.278441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:11.278565 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.279340 (+ 775us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.279514 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":449,"spinlock_wait_cycles":55936,"thread_start_us":129,"threads_started":1}
I20260620 14:26:11.280596 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41443
I20260620 14:26:11.280815 7601 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:11.281109 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40593
I20260620 14:26:11.281188 7603 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:11.281121 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 14:26:11.281327 7602 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:11.281328 7601 heartbeater.cc:511] Master 127.0.98.60:41443 requested a full tablet report, sending...
I20260620 14:26:11.281366 7603 heartbeater.cc:511] Master 127.0.98.61:40593 requested a full tablet report, sending...
I20260620 14:26:11.281517 7602 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260620 14:26:11.282388 7092 ts_manager.cc:194] Registered new tserver with Master: 3f6e991dbbfa482dafbd9c4d9f6809f0 (127.0.98.2:43435)
I20260620 14:26:11.282388 7285 ts_manager.cc:194] Registered new tserver with Master: 3f6e991dbbfa482dafbd9c4d9f6809f0 (127.0.98.2:43435)
I20260620 14:26:11.282437 6499 ts_manager.cc:194] Re-registered known tserver with Master: 3f6e991dbbfa482dafbd9c4d9f6809f0 (127.0.98.2:43435)
I20260620 14:26:11.283236 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60205
I20260620 14:26:11.334525 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.331371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41443 (local address 127.0.0.1:42906)
0620 14:26:11.331504 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.331536 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.331627 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.331920 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.331922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.331939 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:11.332037 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.332043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.332770 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.332774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.333509 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.333517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.334156 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.334159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.334242 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.334263 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.334328 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.334402 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:26:11.334525 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.331439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.0.98.60:41443)
0620 14:26:11.331596 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.331600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.331614 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:26:11.331757 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.331760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.331818 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.331908 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:11.332144 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.332643 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.333632 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.334037 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.334155 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.334303 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.334341 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.334401 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:26:11.335194 7292 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 14:26:11.335538 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef79ad2a6de42ecacfb6489b5381f63. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:11.335681 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: This master's current role is: LEARNER
W20260620 14:26:11.376991 7608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:11.377245 7608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:11.377343 7608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:11.381075 7608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:11.381138 7608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:11.381389 7608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:11.386020 7608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43231
--tserver_master_addrs=127.0.98.62:42021,127.0.98.61:40593,127.0.98.60:41443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:11.386912 7608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:11.387799 7608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:11.395880 7614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:11.395900 7613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:11.395951 7608 server_base.cc:1034] running on GCE node
W20260620 14:26:11.396073 7616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:11.396462 7608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:11.397061 7608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:11.398221 7608 hybrid_clock.cc:648] HybridClock initialized: now 1781965571398214 us; error 31 us; skew 500 ppm
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:11.400501 7608 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:26:11.401337 7608 webserver.cc:460] Webserver started at http://127.0.98.3:43231/ using document root <none> and password file <none>
I20260620 14:26:11.401818 7608 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:11.401914 7608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:11.405084 7608 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:26:11.406984 7623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:11.408088 7608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:11.408252 7608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "5f3847a6bab749679b2632fda04b6f3c"
format_stamp: "Formatted at 2026-06-20 14:26:10 on dist-test-slave-pk3j"
server_key: "adf0782a1ca9c0b8f4b4c423d83f8fe6"
server_key_iv: "ee2211398cd19f59b1dac5799dd7561e"
server_key_version: "encryptionkey@0"
I20260620 14:26:11.408703 7608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:11.417598 7608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:11.419909 7608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:11.420092 7608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:11.420511 7608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:11.421168 7608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:11.421255 7608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:11.421310 7608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:11.421332 7608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:11.429955 7608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37545
I20260620 14:26:11.429975 7736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37545 every 8 connection(s)
I20260620 14:26:11.430892 7608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:11.437523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 7608
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:11.446250 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.434227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36723 (local address 127.0.98.60:41443)
0620 14:26:11.434345 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.434349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.434475 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:26:11.435146 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.435148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.435185 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.435268 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.436083 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.436602 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.438281 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.438614 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.438691 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.442212 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.442218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.442222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.442250 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.443872 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.444496 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.444499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.444501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.444594 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.445032 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.445035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.445036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.445273 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.445335 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.445980 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.446089 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:26:11.447356 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.432955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:43913)
0620 14:26:11.433838 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.433850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.434525 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.435000 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.435005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.435553 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:11.435953 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.435961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.437009 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.437012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.438151 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.438158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.438842 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.438846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.439101 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.440107 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:11.440121 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:11.442448 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:11.444612 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.444617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.444622 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.444913 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.445255 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.445258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.445260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.445346 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.445835 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:11.445841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:11.445927 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.446028 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.446364 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":273,"spinlock_wait_cycles":56576,"thread_start_us":203,"threads_started":1}
I20260620 14:26:11.447353 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.434080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41443 (local address 127.0.98.3:36723)
0620 14:26:11.434387 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.434400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.434534 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.435282 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.435284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.435526 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:11.435942 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.435956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.436736 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.436740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.438148 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.438158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.438842 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.438846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.439116 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.439761 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:11.439786 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:11.442012 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:11.443993 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.443999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.444011 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.444391 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.444717 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.444720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.444722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.444919 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.445453 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:11.445459 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:11.445695 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.446471 (+ 776us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.446602 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":140,"spinlock_wait_cycles":4864,"thread_start_us":119,"threads_started":1}
I20260620 14:26:11.448585 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41443
I20260620 14:26:11.448798 7737 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:11.448930 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.432681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43913 (local address 127.0.98.62:42021)
0620 14:26:11.432804 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.432813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.433920 (+ 1107us) server_negotiation.cc:407] Connection header received
0620 14:26:11.434733 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.434736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.434794 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.434863 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.436106 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.436901 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.438282 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.438624 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.438708 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.442594 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.442601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.442604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.442629 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.444444 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.445039 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.445046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.445049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.445127 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.445457 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.445461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.445464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.445685 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.445779 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.448652 (+ 2873us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.448772 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47,"spinlock_wait_cycles":3456}
I20260620 14:26:11.449734 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260620 14:26:11.449774 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.439709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.0.98.62:42021)
0620 14:26:11.440501 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.440505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.440515 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:26:11.440680 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.440684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.440726 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.440806 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.441774 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.442544 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.443710 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.444068 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.444170 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.445912 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.445915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.445918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.445936 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.447633 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.448204 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.448208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.448210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.448296 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.448588 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.448592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.448593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.448782 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.448894 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.449459 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.449608 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":65,"threads_started":1}
I20260620 14:26:11.449819 7738 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:11.450178 7738 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
Jun 20 14:26:11 dist-test-slave-pk3j krb5kdc[6456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965571, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 14:26:11.451176 6501 ts_manager.cc:194] Re-registered known tserver with Master: 5f3847a6bab749679b2632fda04b6f3c (127.0.98.3:37545)
I20260620 14:26:11.451371 7737 heartbeater.cc:511] Master 127.0.98.60:41443 requested a full tablet report, sending...
I20260620 14:26:11.451896 6501 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43913
I20260620 14:26:11.452049 7285 ts_manager.cc:194] Registered new tserver with Master: 5f3847a6bab749679b2632fda04b6f3c (127.0.98.3:37545)
I20260620 14:26:11.454874 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.432955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40593 (local address 127.0.98.3:53779)
0620 14:26:11.433496 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.433531 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.434525 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.435096 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.435098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.435553 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:11.435955 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.435961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.437043 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.437046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.438171 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.438183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.438883 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.438886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.439450 (+ 564us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.449603 (+ 10153us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:11.449615 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:11.451467 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:11.453346 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.453350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.453351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.453568 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.453894 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:11.453897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:11.453899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:11.453978 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:11.454350 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:11.454353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:11.454438 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.454541 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.454675 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":129,"spinlock_wait_cycles":2816,"thread_start_us":654,"threads_started":1}
I20260620 14:26:11.454842 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.432540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53779 (local address 127.0.98.61:40593)
0620 14:26:11.432655 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.432659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.433632 (+ 973us) server_negotiation.cc:407] Connection header received
0620 14:26:11.434960 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.434963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.435002 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.435057 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.436106 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.436900 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.438316 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.438629 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.438706 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.451647 (+ 12941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.451651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.451653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.451676 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.453232 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.453674 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.453677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.453679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.453788 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.454081 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.454084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.454086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.454255 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.454358 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.454603 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.454708 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:26:11.455387 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40593
I20260620 14:26:11.455463 7739 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:11.455619 7739 heartbeater.cc:511] Master 127.0.98.61:40593 requested a full tablet report, sending...
I20260620 14:26:11.456326 7092 ts_manager.cc:194] Registered new tserver with Master: 5f3847a6bab749679b2632fda04b6f3c (127.0.98.3:37545)
I20260620 14:26:11.453564 6501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:26:11.458105 6501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:26:11.458273 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.440694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.0.98.61:40593)
0620 14:26:11.441033 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.441037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.441050 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:11.441082 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.441085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.441127 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.441207 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.443420 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.444160 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.446650 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.446990 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.447083 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.451078 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.451081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.451083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.451106 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.456126 (+ 5020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.456864 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.456867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.456869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.456950 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.457447 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.457450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.457451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.457644 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.457778 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.458001 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.458108 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":86,"threads_started":1}
I20260620 14:26:11.459120 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.439908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.0.98.60:41443)
0620 14:26:11.440618 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.440630 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.440782 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:26:11.440910 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.440913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.440955 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.441453 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.442566 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.443302 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.446191 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.446531 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.448721 (+ 2190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.448830 (+ 109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.448832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.448835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.448850 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.455659 (+ 6809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.457236 (+ 1577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.457239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.457241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.457323 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.457779 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.457782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.457784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.457963 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.458761 (+ 798us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.458831 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.458935 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":83,"threads_started":1}
I20260620 14:26:11.460592 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: true } } }
I20260620 14:26:11.460711 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:11.463984 7251 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63: attempting to promote NON_VOTER a7ebe6ca063b4c5b8234e63518c59383 to VOTER
I20260620 14:26:11.464293 7251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } }
I20260620 14:26:11.465317 7100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Refusing update from remote peer 9ef79ad2a6de42ecacfb6489b5381f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:26:11.465677 7755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 14:26:11.466209 7295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 LEARNER]: Refusing update from remote peer 9ef79ad2a6de42ecacfb6489b5381f63: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:26:11.466873 7755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 14:26:11.467811 7756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:11.468219 7100 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:11.468945 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:11.469071 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:11.468961 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:11.469225 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:11.470196 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:11.470315 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:26:11.470350 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:11.470450 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:11.470681 7295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:11.472437 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:11.472536 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:26:11.477753 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.472953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.0.1:46488)
0620 14:26:11.473064 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.473078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.473152 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.473905 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.473910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.473932 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:11.474013 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.474019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.475283 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.475285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.476351 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.476359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.477361 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.477367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.477465 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.477486 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.477560 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.477636 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":23}
I20260620 14:26:11.478305 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.473411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37545 (local address 127.0.0.1:48494)
0620 14:26:11.474135 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.474147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.474257 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.474770 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.474774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.474789 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:11.474892 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.474897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.475812 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.475816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.476757 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.476766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.477958 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.477961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.478054 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.478074 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.478136 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.478199 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":18,"thread_start_us":65,"threads_started":1}
I20260620 14:26:11.478425 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.473134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.0.98.1:45309)
0620 14:26:11.473432 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.473444 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.473466 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:26:11.473538 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.473545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.473758 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.473903 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:11.474118 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.475177 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.476501 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.477179 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.477345 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.477476 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.478175 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.478266 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":135,"threads_started":1}
I20260620 14:26:11.478991 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.473114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43435 (local address 127.0.0.1:52086)
0620 14:26:11.473503 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.473514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.473698 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.474893 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.474895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.474908 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:11.474994 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.474998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.476347 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.476351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.477325 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.477338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.478641 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.478644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.478723 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.478741 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.478825 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.478885 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":75,"threads_started":1}
I20260620 14:26:11.479084 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.473314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.0.98.2:43435)
0620 14:26:11.474487 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.474499 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.474519 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:26:11.474586 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.474592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.474760 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.475129 (+ 369us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:11.475222 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.476238 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.477592 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.478297 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.478454 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.478591 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.478852 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.478941 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":83,"threads_started":1}
I20260620 14:26:11.479458 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.473801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.0.98.3:37545)
0620 14:26:11.474084 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.474098 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.474230 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:26:11.474350 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.474357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.474519 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.474662 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:11.474978 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.475705 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.476877 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.477698 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.477785 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.477899 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.479253 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.479333 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
I20260620 14:26:11.480901 7398 tablet_service.cc:1467] Processing CreateTablet for tablet e912acff6a154913ad6d8cf2666e78e0 (DEFAULT_TABLE table=test-table [id=a2347e804c104310b633884633fac65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:11.481786 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e912acff6a154913ad6d8cf2666e78e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:11.481521 7671 tablet_service.cc:1467] Processing CreateTablet for tablet e912acff6a154913ad6d8cf2666e78e0 (DEFAULT_TABLE table=test-table [id=a2347e804c104310b633884633fac65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:11.481518 7535 tablet_service.cc:1467] Processing CreateTablet for tablet e912acff6a154913ad6d8cf2666e78e0 (DEFAULT_TABLE table=test-table [id=a2347e804c104310b633884633fac65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:11.482427 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e912acff6a154913ad6d8cf2666e78e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:11.482429 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e912acff6a154913ad6d8cf2666e78e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:11.487844 7770 tablet_bootstrap.cc:492] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: Bootstrap starting.
I20260620 14:26:11.489635 7771 tablet_bootstrap.cc:492] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Bootstrap starting.
I20260620 14:26:11.490187 7770 tablet_bootstrap.cc:654] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:11.491160 7770 log.cc:824] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:11.491911 7771 tablet_bootstrap.cc:654] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:11.492664 7771 log.cc:824] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:11.493037 7770 tablet_bootstrap.cc:492] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: No bootstrap required, opened a new log
I20260620 14:26:11.493251 7770 ts_tablet_manager.cc:1397] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:26:11.493273 7772 tablet_bootstrap.cc:492] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: Bootstrap starting.
I20260620 14:26:11.494445 7771 tablet_bootstrap.cc:492] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: No bootstrap required, opened a new log
I20260620 14:26:11.494679 7771 ts_tablet_manager.cc:1397] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:26:11.495412 7772 tablet_bootstrap.cc:654] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:11.496400 7772 log.cc:824] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:11.497876 7770 raft_consensus.cc:348] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:11.498116 7770 raft_consensus.cc:374] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:11.498173 7770 raft_consensus.cc:729] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0452603959bd46158c77c4f7b2f2ddef, State: Initialized, Role: FOLLOWER
I20260620 14:26:11.498330 7772 tablet_bootstrap.cc:492] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: No bootstrap required, opened a new log
I20260620 14:26:11.498479 7772 ts_tablet_manager.cc:1397] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:26:11.498595 7770 consensus_queue.cc:260] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:11.498534 7771 raft_consensus.cc:348] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:11.498757 7771 raft_consensus.cc:374] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:11.498818 7771 raft_consensus.cc:729] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6e991dbbfa482dafbd9c4d9f6809f0, State: Initialized, Role: FOLLOWER
I20260620 14:26:11.499207 7771 consensus_queue.cc:260] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:11.499476 7770 ts_tablet_manager.cc:1428] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:26:11.499752 7465 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
I20260620 14:26:11.500041 7771 ts_tablet_manager.cc:1428] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:26:11.500423 7602 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
I20260620 14:26:11.505842 7772 raft_consensus.cc:348] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:11.506042 7772 raft_consensus.cc:374] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:11.506101 7772 raft_consensus.cc:729] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3847a6bab749679b2632fda04b6f3c, State: Initialized, Role: FOLLOWER
I20260620 14:26:11.506479 7772 consensus_queue.cc:260] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:11.507023 7738 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
I20260620 14:26:11.507220 7772 ts_tablet_manager.cc:1428] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260620 14:26:11.512674 7604 tablet.cc:2369] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:26:11.559752 7467 tablet.cc:2369] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:26:11.627120 7778 raft_consensus.cc:482] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:11.627282 7778 raft_consensus.cc:504] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:11.628409 7778 leader_election.cc:288] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0452603959bd46158c77c4f7b2f2ddef (127.0.98.1:45309), 3f6e991dbbfa482dafbd9c4d9f6809f0 (127.0.98.2:43435)
I20260620 14:26:11.632025 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.628569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.98.3:52301)
0620 14:26:11.628701 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.628715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.628819 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.629163 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.629166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.629181 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:11.629327 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.629332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.630126 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.630129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.630984 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.630993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.631613 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.631616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.631709 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.631730 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.631821 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.631883 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":19}
I20260620 14:26:11.632007 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.628674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52301 (local address 127.0.98.1:45309)
0620 14:26:11.628842 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.628846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.628863 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:26:11.628917 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.628920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.628969 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.629052 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:11.629437 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.629999 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.631103 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.631479 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.631615 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.631672 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.631834 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.631900 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:26:11.632233 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.628876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43435 (local address 127.0.98.3:36371)
0620 14:26:11.629037 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:11.629050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:11.629114 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:11.629371 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:11.629373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:11.629388 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:11.629494 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.629499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.630259 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.630265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.631169 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:11.631177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.631846 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.631849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:11.631946 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.631966 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:26:11.632055 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:26:11.632116 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":31}
I20260620 14:26:11.632244 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.628976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36371 (local address 127.0.98.2:43435)
0620 14:26:11.629116 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.629121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.629138 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:26:11.629191 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.629193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.629280 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.629363 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:11.629606 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.630150 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.631283 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.631740 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.631846 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.631902 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.632072 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.632136 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:26:11.632616 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912acff6a154913ad6d8cf2666e78e0" candidate_uuid: "5f3847a6bab749679b2632fda04b6f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0452603959bd46158c77c4f7b2f2ddef" is_pre_election: true
I20260620 14:26:11.632818 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912acff6a154913ad6d8cf2666e78e0" candidate_uuid: "5f3847a6bab749679b2632fda04b6f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" is_pre_election: true
I20260620 14:26:11.632899 7418 raft_consensus.cc:2454] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f3847a6bab749679b2632fda04b6f3c in term 0.
I20260620 14:26:11.633062 7555 raft_consensus.cc:2454] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f3847a6bab749679b2632fda04b6f3c in term 0.
I20260620 14:26:11.633396 7626 leader_election.cc:302] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0452603959bd46158c77c4f7b2f2ddef, 5f3847a6bab749679b2632fda04b6f3c; no voters:
I20260620 14:26:11.633643 7778 raft_consensus.cc:2781] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:11.633719 7778 raft_consensus.cc:482] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:11.633766 7778 raft_consensus.cc:3037] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:11.634678 7778 raft_consensus.cc:504] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:11.634956 7778 leader_election.cc:288] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [CANDIDATE]: Term 1 election: Requested vote from peers 0452603959bd46158c77c4f7b2f2ddef (127.0.98.1:45309), 3f6e991dbbfa482dafbd9c4d9f6809f0 (127.0.98.2:43435)
I20260620 14:26:11.635267 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912acff6a154913ad6d8cf2666e78e0" candidate_uuid: "5f3847a6bab749679b2632fda04b6f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0"
I20260620 14:26:11.635267 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e912acff6a154913ad6d8cf2666e78e0" candidate_uuid: "5f3847a6bab749679b2632fda04b6f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0452603959bd46158c77c4f7b2f2ddef"
I20260620 14:26:11.635376 7555 raft_consensus.cc:3037] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:11.635377 7418 raft_consensus.cc:3037] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:11.636552 7555 raft_consensus.cc:2454] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3847a6bab749679b2632fda04b6f3c in term 1.
I20260620 14:26:11.636552 7418 raft_consensus.cc:2454] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3847a6bab749679b2632fda04b6f3c in term 1.
I20260620 14:26:11.636826 7626 leader_election.cc:302] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0452603959bd46158c77c4f7b2f2ddef, 5f3847a6bab749679b2632fda04b6f3c; no voters:
I20260620 14:26:11.636971 7778 raft_consensus.cc:2781] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:11.637164 7778 raft_consensus.cc:686] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 1 LEADER]: Becoming Leader. State: Replica: 5f3847a6bab749679b2632fda04b6f3c, State: Running, Role: LEADER
I20260620 14:26:11.637579 7778 consensus_queue.cc:237] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:11.640704 6501 catalog_manager.cc:5495] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f3847a6bab749679b2632fda04b6f3c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5f3847a6bab749679b2632fda04b6f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } health_report { overall_health: HEALTHY } } }
I20260620 14:26:11.663959 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.661082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.0.98.3:37545)
0620 14:26:11.661184 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.661188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.661247 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:26:11.661376 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.661379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.661426 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.661517 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:26:11.661734 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.662164 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.662881 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.663066 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.663135 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.663353 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:26:11.663700 (+ 347us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:26:11.663784 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.663832 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.663872 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:26:11.669651 7555 raft_consensus.cc:1261] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 1 FOLLOWER]: Refusing update from remote peer 5f3847a6bab749679b2632fda04b6f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:11.669720 7418 raft_consensus.cc:1261] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef [term 1 FOLLOWER]: Refusing update from remote peer 5f3847a6bab749679b2632fda04b6f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:11.670331 7779 consensus_queue.cc:1035] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:11.670506 7778 consensus_queue.cc:1035] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:11.679556 7784 mvcc.cc:205] Tried to move back new op lower bound from 7298930981551984640 to 7298930981430263808. Current Snapshot: MvccSnapshot[applied={T|T < 7298930981551984640}]
I20260620 14:26:11.682183 7783 mvcc.cc:205] Tried to move back new op lower bound from 7298930981551984640 to 7298930981430263808. Current Snapshot: MvccSnapshot[applied={T|T < 7298930981551984640}]
I20260620 14:26:11.688051 6501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:11.688220 6501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:11.692240 6501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef79ad2a6de42ecacfb6489b5381f63: Sending DeleteTablet for 3 replicas of tablet e912acff6a154913ad6d8cf2666e78e0
I20260620 14:26:11.693468 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet e912acff6a154913ad6d8cf2666e78e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46488
I20260620 14:26:11.693629 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet e912acff6a154913ad6d8cf2666e78e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48494
I20260620 14:26:11.693621 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet e912acff6a154913ad6d8cf2666e78e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52086
I20260620 14:26:11.694137 7794 tablet_replica.cc:330] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: stopping tablet replica
I20260620 14:26:11.694182 7793 tablet_replica.cc:330] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: stopping tablet replica
I20260620 14:26:11.694422 7793 raft_consensus.cc:2229] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:26:11.694768 7793 raft_consensus.cc:2258] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:11.695027 7792 tablet_replica.cc:330] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: stopping tablet replica
I20260620 14:26:11.694402 7794 raft_consensus.cc:2229] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:11.695896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 6469
I20260620 14:26:11.696568 7794 raft_consensus.cc:2258] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:11.697609 7793 ts_tablet_manager.cc:1905] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:11.697947 7794 ts_tablet_manager.cc:1905] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:11.700203 7793 ts_tablet_manager.cc:1918] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:11.700279 7793 log.cc:1196] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/e912acff6a154913ad6d8cf2666e78e0
I20260620 14:26:11.700574 7793 ts_tablet_manager.cc:1939] T e912acff6a154913ad6d8cf2666e78e0 P 5f3847a6bab749679b2632fda04b6f3c: Deleting consensus metadata
I20260620 14:26:11.701083 7794 ts_tablet_manager.cc:1918] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:11.701164 7794 log.cc:1196] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/e912acff6a154913ad6d8cf2666e78e0
I20260620 14:26:11.701476 7794 ts_tablet_manager.cc:1939] T e912acff6a154913ad6d8cf2666e78e0 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Deleting consensus metadata
I20260620 14:26:11.701610 7792 raft_consensus.cc:2229] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:11.701869 7792 raft_consensus.cc:2258] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:26:11.702978 7625 connection.cc:535] server connection from 127.0.0.1:48494 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 14:26:11.703037 7491 connection.cc:535] server connection from 127.0.0.1:52086 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:26:11.703459 7792 ts_tablet_manager.cc:1905] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:11.705812 7792 ts_tablet_manager.cc:1918] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:11.705893 7792 log.cc:1196] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/e912acff6a154913ad6d8cf2666e78e0
I20260620 14:26:11.706179 7792 ts_tablet_manager.cc:1939] T e912acff6a154913ad6d8cf2666e78e0 P 0452603959bd46158c77c4f7b2f2ddef: Deleting consensus metadata
W20260620 14:26:11.707645 7352 connection.cc:441] server connection from 127.0.0.1:46488 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46488 (request call id 1) could send its response
I20260620 14:26:11.717387 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.706164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.0.98.61:40593)
0620 14:26:11.706277 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.706281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.707320 (+ 1039us) server_negotiation.cc:407] Connection header received
0620 14:26:11.707475 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.707480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.707527 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.707580 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.708412 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.709235 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.710684 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.711002 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.711068 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.713486 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.713490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.713493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.713519 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.715388 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.715915 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.715918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.715920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.715994 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.716300 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.716304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.716305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.716541 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.716643 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.717034 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.717150 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:26:11.717411 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.695287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.0.98.60:41443)
0620 14:26:11.695395 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.695399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.706579 (+ 11180us) server_negotiation.cc:407] Connection header received
0620 14:26:11.706779 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.706782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.706831 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.706914 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.708519 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.709300 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.710739 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.711081 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.711148 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.713285 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.713288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.713290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.713314 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.715607 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.716064 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.716070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.716073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.716147 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.716455 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.716458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.716460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.716670 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.716775 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.717121 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.717260 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260620 14:26:11.717634 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:11.706596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.0.98.60:41443)
0620 14:26:11.706706 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:11.706712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:11.706992 (+ 280us) server_negotiation.cc:407] Connection header received
0620 14:26:11.707139 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:11.707143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:11.707183 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:11.707250 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:11.708596 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.709322 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.710706 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:11.711300 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:11.711382 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:11.713726 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:11.713731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:11.713737 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:11.713765 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:11.715805 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.716232 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.716238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.716243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.716341 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:11.716676 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:11.716679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:11.716681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:11.716879 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:11.716983 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:11.717359 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 14:26:11.717463 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":293,"server-negotiator.queue_time_us":40}
I20260620 14:26:12.682132 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:12.681654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:35089)
0620 14:26:12.681918 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:12.682019 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
W20260620 14:26:12.682559 7465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42021 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
I20260620 14:26:12.684917 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:12.684453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:37601)
0620 14:26:12.684768 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:12.684845 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":56,"threads_started":1}
W20260620 14:26:12.685180 7738 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42021 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
I20260620 14:26:12.685413 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:12.684738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:49299)
0620 14:26:12.685252 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:12.685334 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":58,"threads_started":1}
W20260620 14:26:12.685648 7602 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42021 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
I20260620 14:26:13.192410 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ef79ad2a6de42ecacfb6489b5381f63)
I20260620 14:26:13.192625 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } }
I20260620 14:26:13.193090 7807 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ef79ad2a6de42ecacfb6489b5381f63)
I20260620 14:26:13.193271 7807 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } }
I20260620 14:26:13.193786 7806 leader_election.cc:288] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021), b3f1103c8e73480b8b6ffd0cc343f76a (127.0.98.61:40593)
I20260620 14:26:13.194154 7807 leader_election.cc:288] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021), a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60:41443)
I20260620 14:26:13.194592 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.193849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54864)
0620 14:26:13.194127 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.194495 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":76,"threads_started":1}
W20260620 14:26:13.194774 7269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
I20260620 14:26:13.195515 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.194300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54880)
0620 14:26:13.195151 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.195427 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":61,"threads_started":1}
I20260620 14:26:13.195572 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.195338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54894)
0620 14:26:13.195454 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.195512 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 14:26:13.195660 7082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
W20260620 14:26:13.195968 7269 leader_election.cc:334] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
I20260620 14:26:13.196396 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.196213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54896)
0620 14:26:13.196303 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.196351 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":25}
W20260620 14:26:13.196590 7082 leader_election.cc:334] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
I20260620 14:26:13.198447 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.194294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41443 (local address 127.0.0.1:42934)
0620 14:26:13.194736 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.194750 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:13.194840 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:13.195143 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:13.195148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:13.195167 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:13.195304 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:13.195310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:13.196166 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:13.196171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:13.197455 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:13.197474 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:13.198127 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:13.198133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:13.198200 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:13.198214 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:26:13.198262 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:26:13.198330 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":21,"thread_start_us":61,"threads_started":1}
I20260620 14:26:13.198534 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.193974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40593 (local address 127.0.0.1:59904)
0620 14:26:13.194248 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.194264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:13.194354 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:13.194953 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:13.194955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:13.194991 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:13.195106 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:13.195114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:13.196229 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:13.196233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:13.197455 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:13.197465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:13.198066 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:13.198072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:13.198246 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:13.198265 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:26:13.198335 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:26:13.198416 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":71,"spinlock_wait_cycles":7552,"thread_start_us":52,"threads_started":1}
I20260620 14:26:13.198541 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.194420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.0.98.60:41443)
0620 14:26:13.194578 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:13.194582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:13.194803 (+ 221us) server_negotiation.cc:407] Connection header received
0620 14:26:13.194947 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:13.194949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:13.195006 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:13.195070 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:13.195407 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:13.196078 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:13.197608 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:13.198016 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:13.198115 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:13.198272 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:13.198304 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 14:26:13.198396 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":113,"spinlock_wait_cycles":19968,"thread_start_us":48,"threads_started":1}
I20260620 14:26:13.198741 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.194102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.0.98.61:40593)
0620 14:26:13.194752 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:13.194758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:13.194772 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:26:13.194815 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:13.194817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:13.194872 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:13.194951 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:13.195211 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:13.196114 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:13.197563 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:13.197954 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:13.198038 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:13.198111 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:13.198359 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:26:13.198415 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":386,"spinlock_wait_cycles":3456,"thread_start_us":60,"threads_started":1}
I20260620 14:26:13.199106 7108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7ebe6ca063b4c5b8234e63518c59383" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" is_pre_election: true
I20260620 14:26:13.199117 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a7ebe6ca063b4c5b8234e63518c59383" is_pre_election: true
I20260620 14:26:13.199293 7295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f1103c8e73480b8b6ffd0cc343f76a in term 1.
I20260620 14:26:13.199293 7108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ebe6ca063b4c5b8234e63518c59383 in term 1.
I20260620 14:26:13.199575 7084 leader_election.cc:302] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a7ebe6ca063b4c5b8234e63518c59383, b3f1103c8e73480b8b6ffd0cc343f76a; no voters: 9ef79ad2a6de42ecacfb6489b5381f63
I20260620 14:26:13.199597 7272 leader_election.cc:302] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a7ebe6ca063b4c5b8234e63518c59383, b3f1103c8e73480b8b6ffd0cc343f76a; no voters: 9ef79ad2a6de42ecacfb6489b5381f63
I20260620 14:26:13.199751 7807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:26:13.199806 7807 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ef79ad2a6de42ecacfb6489b5381f63)
I20260620 14:26:13.199842 7807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:26:13.199923 7806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:26:13.199987 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ef79ad2a6de42ecacfb6489b5381f63)
I20260620 14:26:13.200047 7806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:26:13.201025 7807 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } }
I20260620 14:26:13.201025 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } }
I20260620 14:26:13.201370 7807 leader_election.cc:288] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 2 election: Requested vote from peers 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021), a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60:41443)
I20260620 14:26:13.201370 7806 leader_election.cc:288] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [CANDIDATE]: Term 2 election: Requested vote from peers 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021), b3f1103c8e73480b8b6ffd0cc343f76a (127.0.98.61:40593)
I20260620 14:26:13.201694 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.201465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54906)
0620 14:26:13.201548 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.201639 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 14:26:13.201711 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a7ebe6ca063b4c5b8234e63518c59383"
I20260620 14:26:13.201835 7295 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b3f1103c8e73480b8b6ffd0cc343f76a in current term 2: Already voted for candidate a7ebe6ca063b4c5b8234e63518c59383 in this term.
I20260620 14:26:13.201853 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.201682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54908)
0620 14:26:13.201758 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.201798 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 14:26:13.201886 7108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7ebe6ca063b4c5b8234e63518c59383" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a"
I20260620 14:26:13.202029 7108 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a7ebe6ca063b4c5b8234e63518c59383 in current term 2: Already voted for candidate b3f1103c8e73480b8b6ffd0cc343f76a in this term.
I20260620 14:26:13.202491 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.202311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54918)
0620 14:26:13.202398 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.202440 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:26:13.202584 7082 leader_election.cc:334] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
I20260620 14:26:13.202637 7082 leader_election.cc:302] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3f1103c8e73480b8b6ffd0cc343f76a; no voters: 9ef79ad2a6de42ecacfb6489b5381f63, a7ebe6ca063b4c5b8234e63518c59383
I20260620 14:26:13.202759 7807 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 14:26:13.202870 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.202660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:54924)
0620 14:26:13.202747 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.202793 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:26:13.202986 7269 leader_election.cc:334] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
I20260620 14:26:13.203048 7269 leader_election.cc:302] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a7ebe6ca063b4c5b8234e63518c59383; no voters: 9ef79ad2a6de42ecacfb6489b5381f63, b3f1103c8e73480b8b6ffd0cc343f76a
I20260620 14:26:13.203198 7806 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 14:26:13.683565 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.683093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:58571)
0620 14:26:13.683350 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.683432 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":91,"threads_started":1}
I20260620 14:26:13.686098 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.685689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:49685)
0620 14:26:13.685958 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.686012 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260620 14:26:13.686548 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:13.686061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:43881)
0620 14:26:13.686386 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:13.686436 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":81,"threads_started":1}
I20260620 14:26:14.684692 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.684226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:48273)
0620 14:26:14.684501 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.684575 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":69,"threads_started":1}
I20260620 14:26:14.687086 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.686668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:36691)
0620 14:26:14.686933 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.686997 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":59,"threads_started":1}
I20260620 14:26:14.687554 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.687154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:34461)
0620 14:26:14.687418 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.687471 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":58,"threads_started":1}
I20260620 14:26:14.736626 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:14.736766 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } }
I20260620 14:26:14.737185 7822 leader_election.cc:288] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021), a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60:41443)
I20260620 14:26:14.737664 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.737302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:55652)
0620 14:26:14.737515 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.737576 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260620 14:26:14.737701 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a7ebe6ca063b4c5b8234e63518c59383" is_pre_election: true
I20260620 14:26:14.737843 7295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f1103c8e73480b8b6ffd0cc343f76a in term 2.
I20260620 14:26:14.738154 7084 leader_election.cc:302] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ebe6ca063b4c5b8234e63518c59383, b3f1103c8e73480b8b6ffd0cc343f76a; no voters:
I20260620 14:26:14.738340 7822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 14:26:14.738416 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:14.738488 7822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:26:14.738783 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.738544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:55654)
0620 14:26:14.738670 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.738721 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:26:14.738899 7082 leader_election.cc:334] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
I20260620 14:26:14.739568 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } }
I20260620 14:26:14.739892 7822 leader_election.cc:288] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 3 election: Requested vote from peers 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021), a7ebe6ca063b4c5b8234e63518c59383 (127.0.98.60:41443)
I20260620 14:26:14.740175 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.739982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:55658)
0620 14:26:14.740064 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.740112 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 14:26:14.740185 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a7ebe6ca063b4c5b8234e63518c59383"
I20260620 14:26:14.740293 7295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:26:14.740821 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.740654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:55670)
0620 14:26:14.740742 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.740780 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:26:14.741050 7082 leader_election.cc:334] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
I20260620 14:26:14.741468 7295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f1103c8e73480b8b6ffd0cc343f76a in term 3.
I20260620 14:26:14.741708 7084 leader_election.cc:302] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a7ebe6ca063b4c5b8234e63518c59383, b3f1103c8e73480b8b6ffd0cc343f76a; no voters: 9ef79ad2a6de42ecacfb6489b5381f63
I20260620 14:26:14.741875 7822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:26:14.742077 7822 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [term 3 LEADER]: Becoming Leader. State: Replica: b3f1103c8e73480b8b6ffd0cc343f76a, State: Running, Role: LEADER
I20260620 14:26:14.742333 7822 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } }
I20260620 14:26:14.743717 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f1103c8e73480b8b6ffd0cc343f76a. Latest consensus state: current_term: 3 leader_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:14.743819 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:14.744091 7827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:14.746206 7827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:14.746608 7827 catalog_manager.cc:1259] Loaded cluster ID: 6c8df4cacf1b4e0d8cc811934d63ba7c
I20260620 14:26:14.746654 7827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:14.747196 7827 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:14.747573 7827 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Loaded TSK: 0
I20260620 14:26:14.748286 7827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:26:14.821905 7098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:26:14.824880 7098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:26:14.827801 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.827555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:55696)
0620 14:26:14.827651 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.827723 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:26:14.827875 7295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [term 3 FOLLOWER]: Refusing update from remote peer b3f1103c8e73480b8b6ffd0cc343f76a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 14:26:14.828267 7825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 14:26:14.828426 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.828247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.0.1:55702)
0620 14:26:14.828320 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.828363 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 14:26:14.828565 7082 consensus_peers.cc:487] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a -> Peer 9ef79ad2a6de42ecacfb6489b5381f63 (127.0.98.62:42021): Couldn't send request to peer 9ef79ad2a6de42ecacfb6489b5381f63. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:26:14.829605 7829 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f1103c8e73480b8b6ffd0cc343f76a. Latest consensus state: current_term: 3 leader_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:14.829730 7829 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:26:14.830685 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:14.830787 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:14.830936 7829 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ef79ad2a6de42ecacfb6489b5381f63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } peers { permanent_uuid: "b3f1103c8e73480b8b6ffd0cc343f76a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40593 } attrs { promote: false } } peers { permanent_uuid: "a7ebe6ca063b4c5b8234e63518c59383" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41443 } attrs { promote: false } } }
I20260620 14:26:14.831101 7829 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ebe6ca063b4c5b8234e63518c59383 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:26:14.840236 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.836236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37545 (local address 127.0.0.1:44626)
0620 14:26:14.836444 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.836456 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:14.836534 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:14.837330 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:14.837333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:14.837351 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:14.837439 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:14.837444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.838293 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.838298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:14.839067 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:14.839075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.839882 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.839887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:14.839972 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.840015 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:26:14.840062 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:26:14.840124 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":26,"thread_start_us":91,"threads_started":1}
I20260620 14:26:14.840328 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.835817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.0.98.2:43435)
0620 14:26:14.836487 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:14.836493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:14.836513 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:26:14.836594 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:14.836598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:14.836651 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:14.836931 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:14.837036 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.837824 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.839262 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.839837 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.839928 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.839985 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:14.840144 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:26:14.840212 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":102,"threads_started":1}
I20260620 14:26:14.840325 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.835718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43435 (local address 127.0.0.1:33544)
0620 14:26:14.836081 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.836095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:14.836182 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:14.836760 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:14.836762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:14.836774 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:14.836834 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:14.836839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.837918 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.837920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:14.839164 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:14.839173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.839925 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.839928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:14.840022 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.840040 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:26:14.840123 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:26:14.840184 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":127}
I20260620 14:26:14.840912 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.836853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.0.98.3:37545)
0620 14:26:14.837096 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:14.837100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:14.837112 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:26:14.837156 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:14.837159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:14.837235 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:14.837319 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:14.837551 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.838183 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.839181 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.839753 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.840620 (+ 867us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.840688 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:14.840720 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 14:26:14.840779 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260620 14:26:14.840978 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1380e24acf4945a44aed5f40968052 (DEFAULT_TABLE table=test-table [id=be1aee2678824ecaa30e1a7886a12aa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:14.841040 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.835928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.0.1:51150)
0620 14:26:14.836151 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.836293 (+ 142us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:14.836386 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:14.836697 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:14.836703 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:14.836723 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:14.836826 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:14.836832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.837783 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.837789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:14.839168 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:14.839173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.840736 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.840739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:14.840812 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.840831 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:26:14.840873 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 14:26:14.840930 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":93,"thread_start_us":102,"threads_started":1}
I20260620 14:26:14.841262 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1380e24acf4945a44aed5f40968052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:14.841426 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1380e24acf4945a44aed5f40968052 (DEFAULT_TABLE table=test-table [id=be1aee2678824ecaa30e1a7886a12aa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:14.841526 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.836055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.0.98.1:45309)
0620 14:26:14.836277 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:14.836282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:14.836467 (+ 185us) server_negotiation.cc:407] Connection header received
0620 14:26:14.836523 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:14.836526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:14.836596 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:14.836694 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:14.836943 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.837675 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.840037 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.840621 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.841239 (+ 618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.841295 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:14.841327 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 14:26:14.841402 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260620 14:26:14.841697 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1380e24acf4945a44aed5f40968052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:14.842077 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1380e24acf4945a44aed5f40968052 (DEFAULT_TABLE table=test-table [id=be1aee2678824ecaa30e1a7886a12aa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:14.842305 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1380e24acf4945a44aed5f40968052. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:14.843703 7839 tablet_bootstrap.cc:492] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Bootstrap starting.
I20260620 14:26:14.844115 7840 tablet_bootstrap.cc:492] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: Bootstrap starting.
I20260620 14:26:14.844612 7841 tablet_bootstrap.cc:492] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef: Bootstrap starting.
I20260620 14:26:14.844870 7839 tablet_bootstrap.cc:654] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:14.844928 7840 tablet_bootstrap.cc:654] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:14.845386 7841 tablet_bootstrap.cc:654] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:14.846027 7840 tablet_bootstrap.cc:492] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: No bootstrap required, opened a new log
I20260620 14:26:14.846149 7840 ts_tablet_manager.cc:1397] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:14.846575 7840 raft_consensus.cc:348] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:14.846705 7839 tablet_bootstrap.cc:492] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0: No bootstrap required, opened a new log
I20260620 14:26:14.846728 7840 raft_consensus.cc:374] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:14.846807 7839 ts_tablet_manager.cc:1397] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:26:14.846798 7840 raft_consensus.cc:729] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3847a6bab749679b2632fda04b6f3c, State: Initialized, Role: FOLLOWER
I20260620 14:26:14.846836 7841 tablet_bootstrap.cc:492] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef: No bootstrap required, opened a new log
I20260620 14:26:14.846920 7841 ts_tablet_manager.cc:1397] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:14.846942 7840 consensus_queue.cc:260] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:14.847294 7840 ts_tablet_manager.cc:1428] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:14.847370 7839 raft_consensus.cc:348] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:14.847404 7841 raft_consensus.cc:348] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:14.847515 7839 raft_consensus.cc:374] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:14.847903 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.847685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:42963)
0620 14:26:14.847777 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.847836 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 14:26:14.847886 7839 raft_consensus.cc:729] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6e991dbbfa482dafbd9c4d9f6809f0, State: Initialized, Role: FOLLOWER
I20260620 14:26:14.848230 7839 consensus_queue.cc:260] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:14.848541 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.848262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:51957)
0620 14:26:14.848389 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.848438 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 14:26:14.848644 7839 ts_tablet_manager.cc:1428] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:14.849435 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.849053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:40009)
0620 14:26:14.849318 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.849369 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260620 14:26:14.849462 7739 heartbeater.cc:503] Master 127.0.98.61:40593 was elected leader, sending a full tablet report...
I20260620 14:26:14.847594 7841 raft_consensus.cc:374] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:14.849907 7841 raft_consensus.cc:729] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0452603959bd46158c77c4f7b2f2ddef, State: Initialized, Role: FOLLOWER
I20260620 14:26:14.850071 7841 consensus_queue.cc:260] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:14.850469 7841 ts_tablet_manager.cc:1428] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:26:14.851071 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.850805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:41445)
0620 14:26:14.850951 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.851002 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 14:26:14.851881 7466 heartbeater.cc:503] Master 127.0.98.61:40593 was elected leader, sending a full tablet report...
I20260620 14:26:14.852329 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.851790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:56981)
0620 14:26:14.852224 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.852272 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370}
I20260620 14:26:14.852455 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.852150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:59713)
0620 14:26:14.852351 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.852397 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 14:26:14.852649 7603 heartbeater.cc:503] Master 127.0.98.61:40593 was elected leader, sending a full tablet report...
I20260620 14:26:14.898759 7847 raft_consensus.cc:482] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:14.898857 7847 raft_consensus.cc:504] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:14.899811 7847 leader_election.cc:288] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f6e991dbbfa482dafbd9c4d9f6809f0 (127.0.98.2:43435), 5f3847a6bab749679b2632fda04b6f3c (127.0.98.3:37545)
I20260620 14:26:14.903198 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.900029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43435 (local address 127.0.98.1:55981)
0620 14:26:14.900168 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.900184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:14.900303 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:14.900555 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:14.900558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:14.900574 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:14.900692 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:14.900697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.901395 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.901400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:14.902157 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:14.902165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.902801 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.902804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:14.902904 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.902931 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:26:14.903016 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:26:14.903079 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 14:26:14.903198 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.900126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55981 (local address 127.0.98.2:43435)
0620 14:26:14.900261 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:14.900266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:14.900280 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:26:14.900399 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:14.900404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:14.900457 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:14.900528 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:14.900805 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.901300 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.902267 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.902662 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.902772 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.902832 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:14.903031 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 14:26:14.903093 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:26:14.903461 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.900383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58223 (local address 127.0.98.3:37545)
0620 14:26:14.900502 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:14.900506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:14.900638 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:26:14.900790 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:14.900793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:14.900837 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:14.900906 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:14.901145 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.901651 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.902578 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.902961 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.903070 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.903119 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:14.903294 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:26:14.903354 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:26:14.903466 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.900283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37545 (local address 127.0.98.1:58223)
0620 14:26:14.900549 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.900567 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:14.900684 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:14.900925 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:14.900927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:14.900940 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:14.901047 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:14.901051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.901750 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.901756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:14.902446 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:14.902455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.903071 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.903074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:14.903169 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.903189 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:26:14.903281 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:26:14.903341 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
I20260620 14:26:14.903656 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1380e24acf4945a44aed5f40968052" candidate_uuid: "0452603959bd46158c77c4f7b2f2ddef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" is_pre_election: true
I20260620 14:26:14.903787 7555 raft_consensus.cc:2454] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0452603959bd46158c77c4f7b2f2ddef in term 0.
I20260620 14:26:14.903971 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1380e24acf4945a44aed5f40968052" candidate_uuid: "0452603959bd46158c77c4f7b2f2ddef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3847a6bab749679b2632fda04b6f3c" is_pre_election: true
I20260620 14:26:14.904176 7691 raft_consensus.cc:2454] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0452603959bd46158c77c4f7b2f2ddef in term 0.
I20260620 14:26:14.904170 7353 leader_election.cc:302] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0452603959bd46158c77c4f7b2f2ddef, 3f6e991dbbfa482dafbd9c4d9f6809f0; no voters:
I20260620 14:26:14.904402 7847 raft_consensus.cc:2781] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:14.904462 7847 raft_consensus.cc:482] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:14.904520 7847 raft_consensus.cc:3037] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:14.905476 7847 raft_consensus.cc:504] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:14.905773 7847 leader_election.cc:288] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [CANDIDATE]: Term 1 election: Requested vote from peers 3f6e991dbbfa482dafbd9c4d9f6809f0 (127.0.98.2:43435), 5f3847a6bab749679b2632fda04b6f3c (127.0.98.3:37545)
I20260620 14:26:14.906077 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1380e24acf4945a44aed5f40968052" candidate_uuid: "0452603959bd46158c77c4f7b2f2ddef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0"
I20260620 14:26:14.906149 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1380e24acf4945a44aed5f40968052" candidate_uuid: "0452603959bd46158c77c4f7b2f2ddef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3847a6bab749679b2632fda04b6f3c"
I20260620 14:26:14.906210 7555 raft_consensus.cc:3037] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:14.906265 7691 raft_consensus.cc:3037] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:14.907040 7555 raft_consensus.cc:2454] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0452603959bd46158c77c4f7b2f2ddef in term 1.
I20260620 14:26:14.907233 7691 raft_consensus.cc:2454] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0452603959bd46158c77c4f7b2f2ddef in term 1.
I20260620 14:26:14.907325 7353 leader_election.cc:302] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0452603959bd46158c77c4f7b2f2ddef, 3f6e991dbbfa482dafbd9c4d9f6809f0; no voters:
I20260620 14:26:14.907521 7847 raft_consensus.cc:2781] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:14.907739 7847 raft_consensus.cc:686] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 1 LEADER]: Becoming Leader. State: Replica: 0452603959bd46158c77c4f7b2f2ddef, State: Running, Role: LEADER
I20260620 14:26:14.907976 7847 consensus_queue.cc:237] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } } peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } }
I20260620 14:26:14.908335 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.908144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:39655)
0620 14:26:14.908233 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.908279 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:26:14.910251 7098 catalog_manager.cc:5495] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0452603959bd46158c77c4f7b2f2ddef (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0452603959bd46158c77c4f7b2f2ddef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0452603959bd46158c77c4f7b2f2ddef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 } health_report { overall_health: UNKNOWN } } }
W20260620 14:26:14.934928 7740 tablet.cc:2369] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:26:14.935371 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.932443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.0.98.1:45309)
0620 14:26:14.932547 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:14.932551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:14.932565 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:26:14.932704 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:14.932706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:14.932749 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:14.932843 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:26:14.933018 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.933511 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.934196 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:14.934373 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:14.934458 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:14.934648 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:26:14.935054 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:26:14.935162 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:14.935235 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:26:14.935274 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:26:14.937834 7555 raft_consensus.cc:1261] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 1 FOLLOWER]: Refusing update from remote peer 0452603959bd46158c77c4f7b2f2ddef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:14.938068 7691 raft_consensus.cc:1261] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c [term 1 FOLLOWER]: Refusing update from remote peer 0452603959bd46158c77c4f7b2f2ddef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:14.938314 7847 consensus_queue.cc:1035] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6e991dbbfa482dafbd9c4d9f6809f0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:14.938552 7849 consensus_queue.cc:1035] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3847a6bab749679b2632fda04b6f3c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:14.939953 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.939722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:36565)
0620 14:26:14.939845 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.939897 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 14:26:14.942768 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.942529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:58103)
0620 14:26:14.942660 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.942709 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 14:26:14.943491 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.943246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:41337)
0620 14:26:14.943391 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.943435 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 14:26:14.944782 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.944491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:52547)
0620 14:26:14.944679 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.944729 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 14:26:14.946426 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.946164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:48275)
0620 14:26:14.946307 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.946358 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 14:26:14.946668 7855 mvcc.cc:205] Tried to move back new op lower bound from 7298930994940678144 to 7298930994825412608. Current Snapshot: MvccSnapshot[applied={T|T < 7298930994940678144}]
I20260620 14:26:14.947235 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.947028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:42023)
0620 14:26:14.947129 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.947175 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:26:14.953169 7098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:14.953331 7098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:14.956910 7098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3f1103c8e73480b8b6ffd0cc343f76a: Sending DeleteTablet for 3 replicas of tablet 9f1380e24acf4945a44aed5f40968052
I20260620 14:26:14.957484 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1380e24acf4945a44aed5f40968052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:14 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33544
I20260620 14:26:14.957576 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1380e24acf4945a44aed5f40968052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:14 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:51150
I20260620 14:26:14.957639 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1380e24acf4945a44aed5f40968052 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:14 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44626
I20260620 14:26:14.957841 7864 tablet_replica.cc:330] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0: stopping tablet replica
I20260620 14:26:14.957916 7866 tablet_replica.cc:330] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: stopping tablet replica
I20260620 14:26:14.957983 7864 raft_consensus.cc:2229] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:14.958061 7866 raft_consensus.cc:2229] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:14.958149 7864 raft_consensus.cc:2258] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:14.958171 7865 tablet_replica.cc:330] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef: stopping tablet replica
I20260620 14:26:14.958222 7866 raft_consensus.cc:2258] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:14.958317 7865 raft_consensus.cc:2229] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:26:14.958552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 7245
I20260620 14:26:14.958599 7865 raft_consensus.cc:2258] T 9f1380e24acf4945a44aed5f40968052 P 0452603959bd46158c77c4f7b2f2ddef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:14.958858 7866 ts_tablet_manager.cc:1905] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:14.958858 7864 ts_tablet_manager.cc:1905] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:14.961378 7864 ts_tablet_manager.cc:1918] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:14.961486 7864 log.cc:1196] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/9f1380e24acf4945a44aed5f40968052
I20260620 14:26:14.961647 7866 ts_tablet_manager.cc:1918] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:14.961725 7866 log.cc:1196] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/9f1380e24acf4945a44aed5f40968052
I20260620 14:26:14.961825 7864 ts_tablet_manager.cc:1939] T 9f1380e24acf4945a44aed5f40968052 P 3f6e991dbbfa482dafbd9c4d9f6809f0: Deleting consensus metadata
I20260620 14:26:14.962003 7866 ts_tablet_manager.cc:1939] T 9f1380e24acf4945a44aed5f40968052 P 5f3847a6bab749679b2632fda04b6f3c: Deleting consensus metadata
I20260620 14:26:14.962440 7084 catalog_manager.cc:4841] TS 3f6e991dbbfa482dafbd9c4d9f6809f0 (127.0.98.2:43435): tablet 9f1380e24acf4945a44aed5f40968052 (table test-table [id=be1aee2678824ecaa30e1a7886a12aa7]) successfully deleted
I20260620 14:26:14.962628 7084 catalog_manager.cc:4841] TS 5f3847a6bab749679b2632fda04b6f3c (127.0.98.3:37545): tablet 9f1380e24acf4945a44aed5f40968052 (table test-table [id=be1aee2678824ecaa30e1a7886a12aa7]) successfully deleted
I20260620 14:26:14.965372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 7472
I20260620 14:26:14.965553 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:14.965288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45309 (local address 127.0.0.1:51164)
0620 14:26:14.965415 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:14.965474 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:26:14.966018 7084 catalog_manager.cc:4584] TS 0452603959bd46158c77c4f7b2f2ddef (127.0.98.1:45309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f1380e24acf4945a44aed5f40968052: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45309: connect: Connection refused (error 111)
I20260620 14:26:14.971824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 7608
I20260620 14:26:14.978065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 6965
I20260620 14:26:14.983350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 7152
2026-06-20T14:26:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6719 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:26:15 dist-test-slave-pk3j krb5kdc[7869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:26:15 dist-test-slave-pk3j krb5kdc[7869](info): set up 2 sockets
Jun 20 14:26:15 dist-test-slave-pk3j krb5kdc[7869](info): commencing operation
krb5kdc: starting...
W20260620 14:26:15.985497 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:26:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 14:26:16.091693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45233
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:16.190999 7882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:16.191191 7882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:16.191243 7882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:16.194885 7882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:16.194921 7882 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:26:16.194936 7882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:16.194969 7882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:16.194984 7882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:16.194998 7882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:16.199757 7882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45233
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:16.200501 7882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:16.201301 7882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:16.208890 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:16.208959 7882 server_base.cc:1034] running on GCE node
W20260620 14:26:16.208913 7888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:16.208925 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:16.209535 7882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:16.210389 7882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:16.211572 7882 hybrid_clock.cc:648] HybridClock initialized: now 1781965576211546 us; error 42 us; skew 500 ppm
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:16.214138 7882 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 14:26:16.214852 7882 webserver.cc:460] Webserver started at http://127.0.98.62:36941/ using document root <none> and password file <none>
I20260620 14:26:16.215351 7882 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:16.215440 7882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:16.215654 7882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:16.217468 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "9715a01610ff6f20bd3c3d0db8fdb8f4"
server_key_iv: "0775838a090568d7a81a61610dda7d04"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.217990 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "9715a01610ff6f20bd3c3d0db8fdb8f4"
server_key_iv: "0775838a090568d7a81a61610dda7d04"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.221063 7882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:26:16.223117 7897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:16.224136 7882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:16.224263 7882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "9715a01610ff6f20bd3c3d0db8fdb8f4"
server_key_iv: "0775838a090568d7a81a61610dda7d04"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.224361 7882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:16.238029 7882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:16.240844 7882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:16.241026 7882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:16.247575 7882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45233
I20260620 14:26:16.247581 7959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45233 every 8 connection(s)
I20260620 14:26:16.248353 7882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:16.250847 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:16.256420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 7882
I20260620 14:26:16.256461 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5: Bootstrap starting.
I20260620 14:26:16.256551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:16.256844 392 external_mini_cluster.cc:1428] Setting key bd3f8a3c3ad5450a9716172792d792de
I20260620 14:26:16.258932 7960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:16.259730 7960 log.cc:824] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:16.261664 7960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5: No bootstrap required, opened a new log
I20260620 14:26:16.265173 7960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45233 } }
I20260620 14:26:16.265444 7960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:16.265508 7960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6906fc9a173349bfa33e2f6ea3ff7ce5, State: Initialized, Role: FOLLOWER
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:16.265944 7960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45233 } }
I20260620 14:26:16.266103 7960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:16.266170 7960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:16.266278 7960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:16.267306 7960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45233 } }
I20260620 14:26:16.267640 7960 leader_election.cc:302] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6906fc9a173349bfa33e2f6ea3ff7ce5; no voters:
I20260620 14:26:16.267956 7960 leader_election.cc:288] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:16.268014 7965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:16.268204 7965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 6906fc9a173349bfa33e2f6ea3ff7ce5, State: Running, Role: LEADER
I20260620 14:26:16.268505 7965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45233 } }
I20260620 14:26:16.269030 7960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:16.270083 7966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45233 } } }
I20260620 14:26:16.270201 7966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:16.270457 7967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6906fc9a173349bfa33e2f6ea3ff7ce5. Latest consensus state: current_term: 1 leader_uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906fc9a173349bfa33e2f6ea3ff7ce5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45233 } } }
I20260620 14:26:16.270542 7967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:16.270942 7974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:16.271761 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.258083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.0.98.62:45233)
0620 14:26:16.258487 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.258504 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.258542 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:26:16.259351 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.259382 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.259747 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.260081 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:16.260838 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.262210 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.262992 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.263667 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.263823 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.266292 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:16.266298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:16.266317 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:16.266366 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:16.268215 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.268669 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.268673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.268678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.268786 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.269025 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.269030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.269031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.269530 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:16.270417 (+ 887us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.270562 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.270871 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":100,"threads_started":1}
I20260620 14:26:16.274289 7974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:16.278798 7974 catalog_manager.cc:1347] Generated new cluster ID: 8c3465389e42448e997977ed810a0a5b
I20260620 14:26:16.278853 7974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:16.308323 7974 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:16.309355 7974 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:16.315495 7974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5: Generated new TSK 0
I20260620 14:26:16.316112 7974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 14:26:16.343755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45233
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:16.445653 7986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:16.445858 7986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:16.445904 7986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:16.449801 7986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:16.449846 7986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:16.449888 7986 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:26:16.449919 7986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:16.454655 7986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:16.455408 7986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:16.456205 7986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:16.464810 7992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:16.464856 7994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:16.464983 7986 server_base.cc:1034] running on GCE node
W20260620 14:26:16.465133 7991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:16.465507 7986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:16.466042 7986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:16.467212 7986 hybrid_clock.cc:648] HybridClock initialized: now 1781965576467198 us; error 53 us; skew 500 ppm
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:16.470018 7986 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:26:16.470896 7986 webserver.cc:460] Webserver started at http://127.0.98.1:43167/ using document root <none> and password file <none>
I20260620 14:26:16.471401 7986 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:16.471485 7986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:16.471673 7986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:16.473495 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "5e665f1796704b0aae9a3e2f08f507f6"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "0e72c5b190c77138a8da88bb437f6e92"
server_key_iv: "08d8d28b2542302ab3432af51ae6c395"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.474012 7986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5e665f1796704b0aae9a3e2f08f507f6"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "0e72c5b190c77138a8da88bb437f6e92"
server_key_iv: "08d8d28b2542302ab3432af51ae6c395"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.477399 7986 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:26:16.479496 8001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:16.480460 7986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:16.480582 7986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "5e665f1796704b0aae9a3e2f08f507f6"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "0e72c5b190c77138a8da88bb437f6e92"
server_key_iv: "08d8d28b2542302ab3432af51ae6c395"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.480676 7986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:16.497416 7986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:16.499912 7986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:16.500104 7986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:16.501437 7986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:16.501510 7986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:16.501585 7986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:16.501627 7986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:16.511988 7986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37667
I20260620 14:26:16.512044 8116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37667 every 8 connection(s)
I20260620 14:26:16.512804 7986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:26:16.514932 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.502173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60317 (local address 127.0.98.62:45233)
0620 14:26:16.502306 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.502323 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.503041 (+ 718us) server_negotiation.cc:407] Connection header received
0620 14:26:16.504093 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.504097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.504151 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.504205 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:16.505893 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.506782 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.508516 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.508929 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.509055 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.511784 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:16.511791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:16.511797 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:16.511827 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:16.513452 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.513917 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.513920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.513921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.513987 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.514198 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.514199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.514200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.514328 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:16.514399 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.514745 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.514824 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:26:16.515547 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.502521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45233 (local address 127.0.98.1:60317)
0620 14:26:16.502912 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:16.502956 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:16.503822 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:16.504608 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:16.504614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:16.505077 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:16.505704 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.505725 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.506939 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.506942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.508386 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.508398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.509083 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.509089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.509262 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.509802 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:16.509815 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:16.511609 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:16.513541 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.513546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.513557 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.513839 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.514044 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.514045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.514046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.514148 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.514394 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:16.514398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:16.514571 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 14:26:16.514702 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:26:16.514892 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":114,"threads_started":1}
I20260620 14:26:16.516615 8117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45233
I20260620 14:26:16.516757 8117 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:16.517268 8117 heartbeater.cc:511] Master 127.0.98.62:45233 requested a full tablet report, sending...
I20260620 14:26:16.518615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 7986
I20260620 14:26:16.518743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:26:16.518772 7914 ts_manager.cc:194] Registered new tserver with Master: 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667)
I20260620 14:26:16.519037 392 external_mini_cluster.cc:1428] Setting key 2458ef9bbaed5b1282f0a291695544b8
I20260620 14:26:16.520124 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60317
I20260620 14:26:16.528054 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.518693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57433 (local address 127.0.98.62:45233)
0620 14:26:16.518792 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.518795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.520915 (+ 2120us) server_negotiation.cc:407] Connection header received
0620 14:26:16.521091 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.521094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.521136 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.521250 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:16.521789 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.522430 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.523616 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.523800 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.523865 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.525025 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:16.525027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:16.525029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:16.525046 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:16.526343 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.526865 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.526869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.526871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.526945 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.527279 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.527282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.527287 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.527444 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:16.527516 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.527806 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.527934 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:26:16.528220 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.518650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45233 (local address 127.0.98.1:57433)
0620 14:26:16.520837 (+ 2187us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:16.520854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:16.520974 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:16.521291 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:16.521296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:16.521504 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:16.521667 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.521677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.522559 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.522562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.523494 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.523507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.523903 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.523909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.524033 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.524485 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:16.524500 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:16.524911 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:16.526457 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.526461 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.526466 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.526729 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.527036 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.527041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.527044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.527150 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.527546 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:16.527551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:16.527641 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 14:26:16.527756 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:26:16.527908 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2098,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 14:26:16.542007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45233
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:16.642709 8126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:16.642905 8126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:16.642943 8126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:16.646673 8126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:16.646723 8126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:16.646759 8126 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:26:16.646790 8126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:16.651494 8126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:16.652227 8126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:16.653017 8126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:16.661191 8131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:16.661580 8126 server_base.cc:1034] running on GCE node
W20260620 14:26:16.661187 8134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:16.661185 8132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:16.662122 8126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:16.662648 8126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:16.663822 8126 hybrid_clock.cc:648] HybridClock initialized: now 1781965576663801 us; error 39 us; skew 500 ppm
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:16.666291 8126 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:26:16.667052 8126 webserver.cc:460] Webserver started at http://127.0.98.2:44169/ using document root <none> and password file <none>
I20260620 14:26:16.667493 8126 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:16.667546 8126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:16.667702 8126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:16.669490 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "78abda291c3e49089c19e4fa191cd82b"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "6b23a0da3bbcfc8c61dd5e0003ff9dd4"
server_key_iv: "fc99bb116313fa9b7be154356bce36f3"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.669992 8126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78abda291c3e49089c19e4fa191cd82b"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "6b23a0da3bbcfc8c61dd5e0003ff9dd4"
server_key_iv: "fc99bb116313fa9b7be154356bce36f3"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.673353 8126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:26:16.675424 8141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:16.676499 8126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:16.676591 8126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "78abda291c3e49089c19e4fa191cd82b"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "6b23a0da3bbcfc8c61dd5e0003ff9dd4"
server_key_iv: "fc99bb116313fa9b7be154356bce36f3"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.676667 8126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:16.720016 8126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:16.722565 8126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:16.722711 8126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:16.724026 8126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:16.724074 8126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:16.724135 8126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:16.724169 8126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:16.734416 8126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37085
I20260620 14:26:16.734475 8256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37085 every 8 connection(s)
I20260620 14:26:16.735221 8126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:16.736828 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.724814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39131 (local address 127.0.98.62:45233)
0620 14:26:16.724965 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.724972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.725763 (+ 791us) server_negotiation.cc:407] Connection header received
0620 14:26:16.726937 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.726943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.727007 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.727086 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:16.728789 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.729464 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.730467 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.730717 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.730828 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.733542 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:16.733546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:16.733548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:16.733572 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:16.735014 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.735545 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.735549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.735551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.735635 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.735975 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.735977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.735978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.736136 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:16.736210 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.736608 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.736716 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:26:16.736974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 8126
I20260620 14:26:16.737097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:26:16.737417 392 external_mini_cluster.cc:1428] Setting key 41098af01196d6a64bf7742a29d5b7fe
I20260620 14:26:16.737771 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.725117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45233 (local address 127.0.98.2:39131)
0620 14:26:16.725596 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:16.725650 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:16.726684 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:16.727532 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:16.727544 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:16.728020 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:16.728621 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.728640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.729582 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.729585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.730349 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.730357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.730871 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.730883 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.731038 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.731594 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:16.731614 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:16.733377 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:16.735121 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.735126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.735138 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.735420 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.735734 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.735736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.735738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.735872 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.736224 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:16.736228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:16.736408 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 14:26:16.736558 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 14:26:16.736847 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":104,"threads_started":1}
I20260620 14:26:16.738981 8257 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45233
I20260620 14:26:16.739140 8257 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:16.739629 8257 heartbeater.cc:511] Master 127.0.98.62:45233 requested a full tablet report, sending...
I20260620 14:26:16.740567 7914 ts_manager.cc:194] Registered new tserver with Master: 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085)
I20260620 14:26:16.741298 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39131
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:26:16.747200 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.740717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33255 (local address 127.0.98.62:45233)
0620 14:26:16.740893 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.740896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.740983 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:26:16.741130 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.741134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.741180 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.741284 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:16.741678 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.742304 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.743207 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.743428 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.743528 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.744489 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:16.744492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:16.744494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:16.744514 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:16.745713 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.746096 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.746098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.746100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.746170 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.746419 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.746423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.746424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.746603 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:16.746721 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.746961 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.747060 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 14:26:16.747228 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.740672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45233 (local address 127.0.98.2:33255)
0620 14:26:16.740911 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:16.740925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:16.741036 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:16.741296 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:16.741298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:16.741480 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:16.741585 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.741591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.742437 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.742440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.743095 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.743103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.743540 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.743543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.743631 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.744075 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:16.744085 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:16.744370 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:16.745813 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.745817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.745819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.745991 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.746254 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.746257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.746259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.746337 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.746712 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:16.746715 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:16.746801 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 14:26:16.746912 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:26:16.747037 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:26:16.762215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45233
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:16.863838 8266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:16.864037 8266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:16.864075 8266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:16.868104 8266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:16.868146 8266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:16.868198 8266 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:26:16.868237 8266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:16.873183 8266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:16.874030 8266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:16.874830 8266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:16.883322 8271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:16.883445 8266 server_base.cc:1034] running on GCE node
W20260620 14:26:16.883342 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:16.883602 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:16.883962 8266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:16.884557 8266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:16.885702 8266 hybrid_clock.cc:648] HybridClock initialized: now 1781965576885687 us; error 27 us; skew 500 ppm
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:16.888187 8266 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:26:16.888965 8266 webserver.cc:460] Webserver started at http://127.0.98.3:42753/ using document root <none> and password file <none>
I20260620 14:26:16.889499 8266 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:16.889582 8266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:16.889780 8266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:16.891470 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "8b360b362a904967b76e36a73e9ecd32"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "62a4d290bf72de35c4ddd06d29f036c8"
server_key_iv: "795d15129eb7b6585b91dbb89854a560"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.891964 8266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8b360b362a904967b76e36a73e9ecd32"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "62a4d290bf72de35c4ddd06d29f036c8"
server_key_iv: "795d15129eb7b6585b91dbb89854a560"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.895128 8266 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 14:26:16.897228 8281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:16.898310 8266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:26:16.898443 8266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "8b360b362a904967b76e36a73e9ecd32"
format_stamp: "Formatted at 2026-06-20 14:26:16 on dist-test-slave-pk3j"
server_key: "62a4d290bf72de35c4ddd06d29f036c8"
server_key_iv: "795d15129eb7b6585b91dbb89854a560"
server_key_version: "encryptionkey@0"
I20260620 14:26:16.898542 8266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:16.914005 8266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:16.917021 8266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:16.917240 8266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:16.918682 8266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:16.918751 8266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:16.918812 8266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:16.918864 8266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:16.929105 8266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33209
I20260620 14:26:16.929248 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33209 every 8 connection(s)
I20260620 14:26:16.930058 8266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:16.931617 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.919270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38399 (local address 127.0.98.62:45233)
0620 14:26:16.919422 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.919427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.920148 (+ 721us) server_negotiation.cc:407] Connection header received
0620 14:26:16.920998 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.921003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.921061 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.921158 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:16.922287 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.923095 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.924330 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.924558 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.924666 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.928007 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:16.928013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:16.928018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:16.928043 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:16.929701 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.930140 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.930144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.930145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.930231 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.930590 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.930593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.930595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.930815 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:16.930921 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.931411 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.931500 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:26:16.932274 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.919600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45233 (local address 127.0.98.3:38399)
0620 14:26:16.919997 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:16.920041 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:16.920803 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:16.921367 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:16.921378 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:16.921716 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:16.922127 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.922138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.923223 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.923227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.924213 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.924222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.924698 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.924703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.924868 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.925461 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:16.925477 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:16.927771 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:16.929801 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.929805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.929814 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.930029 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.930321 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.930323 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.930324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.930440 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.930917 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:16.930922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:16.931179 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 14:26:16.931361 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 14:26:16.931569 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":118,"threads_started":1}
I20260620 14:26:16.933493 8397 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45233
I20260620 14:26:16.933606 8397 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:16.934031 8397 heartbeater.cc:511] Master 127.0.98.62:45233 requested a full tablet report, sending...
I20260620 14:26:16.934962 7914 ts_manager.cc:194] Registered new tserver with Master: 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209)
I20260620 14:26:16.935657 7914 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38399
I20260620 14:26:16.937062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 8266
I20260620 14:26:16.937191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:26:16.937453 392 external_mini_cluster.cc:1428] Setting key 488ef8ba9558f41feef7fa4703da1ce2
I20260620 14:26:16.939702 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:26:16.942334 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.935298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45233 (local address 127.0.98.3:45105)
0620 14:26:16.935567 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:16.935580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:16.935666 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:16.935869 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:16.935872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:16.936058 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:16.936153 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.936158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.937118 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.937121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.937898 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.937907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.938446 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.938450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.938526 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.939018 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:16.939028 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:16.939285 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:16.940886 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.940892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.940897 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.941146 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.941477 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:16.941483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:16.941487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:16.941582 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:16.941995 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:16.941997 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:16.942061 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:26:16.942137 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:26:16.942226 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":117,"threads_started":1}
I20260620 14:26:16.942513 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.935363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45105 (local address 127.0.98.62:45233)
0620 14:26:16.935478 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.935482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.935646 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:26:16.935742 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.935744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.935775 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.935831 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:16.936234 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.936968 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.938015 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.938330 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.938408 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.939408 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:16.939412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:16.939414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:16.939438 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:16.940782 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.941285 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.941287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.941289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.941357 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.941674 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.941679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.941681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.941904 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:16.942002 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.942214 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.942327 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:26:16.961467 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:16.956105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.0.98.62:45233)
0620 14:26:16.956203 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.956206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.956217 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:26:16.956244 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.956246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.956275 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.956356 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:16.956979 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.957519 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.958301 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.958473 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.958537 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.961309 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45564: BlockingRecv error: recv got EOF from 127.0.0.1:45564 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 14:26:16 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:16.969326 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.962525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.0.98.62:45233)
0620 14:26:16.962626 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.962628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.962760 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:26:16.962831 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.962833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.962860 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.962918 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:16.963378 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.963773 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.964451 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.964627 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.964693 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.966610 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:16.966612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:16.966614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:16.966637 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:16.967986 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.968317 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.968319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.968320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.968382 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:16.968677 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:16.968679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:16.968680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:16.968819 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:16.968909 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.969093 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.969166 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:26:16.971789 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:26:16.974653 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:26:16.987658 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.982286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37085 (local address 127.0.0.1:57922)
0620 14:26:16.982764 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:16.982804 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:16.982912 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:16.983563 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:16.983570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:16.983593 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:16.983716 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.983727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.984895 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.984899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.986128 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.986136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.987177 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.987180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.987289 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.987323 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:26:16.987430 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:26:16.987536 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":70,"threads_started":1}
I20260620 14:26:16.987715 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.982394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.0.98.2:37085)
0620 14:26:16.983047 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.983058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.983079 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:26:16.983171 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.983176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.983374 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.983612 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:16.983822 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.984790 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.986324 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.987043 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.987162 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.987304 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.987470 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.987559 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":69,"threads_started":1}
I20260620 14:26:16.988171 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.982367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37667 (local address 127.0.0.1:57674)
0620 14:26:16.983119 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:16.983132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:16.983199 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:16.984368 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:16.984371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:16.984389 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:16.984464 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.984468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.985747 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.985749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.986797 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.986806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.987820 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.987826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.987918 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.987938 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:26:16.987991 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:26:16.988053 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":98,"threads_started":1}
I20260620 14:26:16.988305 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.982612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33209 (local address 127.0.0.1:54824)
0620 14:26:16.983330 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:16.983342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:16.983419 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:16.984471 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:16.984473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:16.984486 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:16.984587 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.984592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.985583 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.985587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.986655 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:16.986664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.987921 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.987926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:16.988010 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.988027 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:26:16.988117 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:26:16.988191 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":75,"threads_started":1}
I20260620 14:26:16.988379 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.982949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.0.98.3:33209)
0620 14:26:16.983429 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.983441 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.983464 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:26:16.983528 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.983532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.983689 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.983838 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:16.984694 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.985483 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.986822 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.987597 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.987695 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.987846 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.988161 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.988261 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":82,"threads_started":1}
I20260620 14:26:16.989073 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:16.982941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.0.98.1:37667)
0620 14:26:16.984078 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:16.984090 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:16.984105 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:26:16.984157 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:16.984160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:16.984273 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:16.984368 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:16.984590 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.985647 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.986918 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:16.987669 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:16.988677 (+ 1008us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:16.988833 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:16.988880 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:26:16.988953 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":90,"threads_started":1}
I20260620 14:26:16.990645 8191 tablet_service.cc:1467] Processing CreateTablet for tablet b710e07ae358459c997c90cc486cfbf4 (DEFAULT_TABLE table=test-table [id=d84185d123784fc8880991e65ab36dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:16.991123 8051 tablet_service.cc:1467] Processing CreateTablet for tablet b710e07ae358459c997c90cc486cfbf4 (DEFAULT_TABLE table=test-table [id=d84185d123784fc8880991e65ab36dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:16.991559 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b710e07ae358459c997c90cc486cfbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:16.991961 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b710e07ae358459c997c90cc486cfbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:16.995002 8331 tablet_service.cc:1467] Processing CreateTablet for tablet b710e07ae358459c997c90cc486cfbf4 (DEFAULT_TABLE table=test-table [id=d84185d123784fc8880991e65ab36dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:16.995911 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b710e07ae358459c997c90cc486cfbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:16.998587 8425 tablet_bootstrap.cc:492] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: Bootstrap starting.
I20260620 14:26:16.998782 8426 tablet_bootstrap.cc:492] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: Bootstrap starting.
I20260620 14:26:17.000845 8425 tablet_bootstrap.cc:654] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.000845 8426 tablet_bootstrap.cc:654] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.001665 8425 log.cc:824] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:17.001665 8426 log.cc:824] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:17.002256 8427 tablet_bootstrap.cc:492] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: Bootstrap starting.
I20260620 14:26:17.003973 8426 tablet_bootstrap.cc:492] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: No bootstrap required, opened a new log
I20260620 14:26:17.003970 8425 tablet_bootstrap.cc:492] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: No bootstrap required, opened a new log
I20260620 14:26:17.004158 8426 ts_tablet_manager.cc:1397] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 14:26:17.004180 8425 ts_tablet_manager.cc:1397] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 14:26:17.004271 8427 tablet_bootstrap.cc:654] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.005035 8427 log.cc:824] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:17.006860 8427 tablet_bootstrap.cc:492] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: No bootstrap required, opened a new log
I20260620 14:26:17.007027 8427 ts_tablet_manager.cc:1397] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:26:17.007503 8425 raft_consensus.cc:348] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.007719 8425 raft_consensus.cc:374] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.007789 8425 raft_consensus.cc:729] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78abda291c3e49089c19e4fa191cd82b, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.007808 8426 raft_consensus.cc:348] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.008100 8426 raft_consensus.cc:374] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.008165 8426 raft_consensus.cc:729] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e665f1796704b0aae9a3e2f08f507f6, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.008257 8425 consensus_queue.cc:260] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.008541 8426 consensus_queue.cc:260] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.008858 8257 heartbeater.cc:503] Master 127.0.98.62:45233 was elected leader, sending a full tablet report...
I20260620 14:26:17.009217 8425 ts_tablet_manager.cc:1428] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:26:17.009507 8426 ts_tablet_manager.cc:1428] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260620 14:26:17.009785 8117 heartbeater.cc:503] Master 127.0.98.62:45233 was elected leader, sending a full tablet report...
I20260620 14:26:17.010551 8427 raft_consensus.cc:348] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.010826 8427 raft_consensus.cc:374] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.010910 8427 raft_consensus.cc:729] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b360b362a904967b76e36a73e9ecd32, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.011273 8427 consensus_queue.cc:260] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.012178 8427 ts_tablet_manager.cc:1428] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:26:17.012334 8397 heartbeater.cc:503] Master 127.0.98.62:45233 was elected leader, sending a full tablet report...
W20260620 14:26:17.013863 8118 tablet.cc:2369] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:26:17.046142 8433 raft_consensus.cc:482] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:17.046293 8433 raft_consensus.cc:504] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.047179 8433 leader_election.cc:288] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085), 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667)
I20260620 14:26:17.050937 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.047390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37667 (local address 127.0.98.3:53559)
0620 14:26:17.047513 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:17.047529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:17.047624 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:17.047879 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:17.047882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:17.047909 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:17.048017 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.048022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.048878 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.048880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.050022 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.050032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.050622 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.050627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.050721 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.050739 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:26:17.050792 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:26:17.050843 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:26:17.050993 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.047473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53559 (local address 127.0.98.1:37667)
0620 14:26:17.047583 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.047588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.047605 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:26:17.047741 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.047747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.047799 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.047874 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:17.048124 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.048778 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.050139 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.050510 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.050588 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.050669 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.050824 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.050889 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:26:17.051673 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.047390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37085 (local address 127.0.98.3:47701)
0620 14:26:17.047906 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:17.047916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:17.048023 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:17.048275 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:17.048278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:17.048292 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:17.048421 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.048432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.049264 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.049268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.050529 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.050536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.051351 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.051355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.051451 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.051473 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:26:17.051539 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:26:17.051583 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":228,"thread_start_us":112,"threads_started":1}
I20260620 14:26:17.051643 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b710e07ae358459c997c90cc486cfbf4" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6" is_pre_election: true
I20260620 14:26:17.051743 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.047571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47701 (local address 127.0.98.2:37085)
0620 14:26:17.047705 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.047711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.047992 (+ 281us) server_negotiation.cc:407] Connection header received
0620 14:26:17.048119 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.048122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.048167 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.048239 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:17.048565 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.049140 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.050664 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.051217 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.051318 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.051369 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.051580 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.051641 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:26:17.051868 8071 raft_consensus.cc:2454] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b360b362a904967b76e36a73e9ecd32 in term 0.
I20260620 14:26:17.052318 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b710e07ae358459c997c90cc486cfbf4" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78abda291c3e49089c19e4fa191cd82b" is_pre_election: true
I20260620 14:26:17.052326 8284 leader_election.cc:302] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e665f1796704b0aae9a3e2f08f507f6, 8b360b362a904967b76e36a73e9ecd32; no voters:
I20260620 14:26:17.052546 8211 raft_consensus.cc:2454] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b360b362a904967b76e36a73e9ecd32 in term 0.
I20260620 14:26:17.052568 8433 raft_consensus.cc:2781] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:17.052636 8433 raft_consensus.cc:482] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:17.052675 8433 raft_consensus.cc:3037] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.053563 8433 raft_consensus.cc:504] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.053845 8433 leader_election.cc:288] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 election: Requested vote from peers 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085), 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667)
I20260620 14:26:17.054069 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b710e07ae358459c997c90cc486cfbf4" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78abda291c3e49089c19e4fa191cd82b"
I20260620 14:26:17.054171 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b710e07ae358459c997c90cc486cfbf4" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6"
I20260620 14:26:17.054209 8211 raft_consensus.cc:3037] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.054291 8071 raft_consensus.cc:3037] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.055162 8211 raft_consensus.cc:2454] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b360b362a904967b76e36a73e9ecd32 in term 1.
I20260620 14:26:17.055286 8071 raft_consensus.cc:2454] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b360b362a904967b76e36a73e9ecd32 in term 1.
I20260620 14:26:17.055469 8283 leader_election.cc:302] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78abda291c3e49089c19e4fa191cd82b, 8b360b362a904967b76e36a73e9ecd32; no voters:
I20260620 14:26:17.055667 8433 raft_consensus.cc:2781] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:17.055846 8433 raft_consensus.cc:686] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 1 LEADER]: Becoming Leader. State: Replica: 8b360b362a904967b76e36a73e9ecd32, State: Running, Role: LEADER
I20260620 14:26:17.056154 8433 consensus_queue.cc:237] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.059058 7914 catalog_manager.cc:5495] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8b360b362a904967b76e36a73e9ecd32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } health_report { overall_health: HEALTHY } } }
I20260620 14:26:17.082510 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.079267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.0.98.3:33209)
0620 14:26:17.079370 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.079374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.079452 (+ 78us) server_negotiation.cc:407] Connection header received
0620 14:26:17.079638 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.079640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.079683 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.079781 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:26:17.080010 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.080480 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.081317 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.081522 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.081616 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.081869 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:26:17.082190 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:26:17.082301 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.082357 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.082400 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:26:17.087098 8071 raft_consensus.cc:1261] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Refusing update from remote peer 8b360b362a904967b76e36a73e9ecd32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:17.087392 8211 raft_consensus.cc:1261] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Refusing update from remote peer 8b360b362a904967b76e36a73e9ecd32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:17.087666 8433 consensus_queue.cc:1035] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:17.087875 8435 consensus_queue.cc:1035] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:17.094599 8439 mvcc.cc:205] Tried to move back new op lower bound from 7298931003743461376 to 7298931003624861696. Current Snapshot: MvccSnapshot[applied={T|T < 7298931003743461376}]
I20260620 14:26:17.094957 8440 mvcc.cc:205] Tried to move back new op lower bound from 7298931003743461376 to 7298931003624861696. Current Snapshot: MvccSnapshot[applied={T|T < 7298931003743461376}]
I20260620 14:26:17.102344 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:17.102484 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:17.104450 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5: Sending DeleteTablet for 3 replicas of tablet b710e07ae358459c997c90cc486cfbf4
I20260620 14:26:17.105252 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet b710e07ae358459c997c90cc486cfbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57922
I20260620 14:26:17.105469 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet b710e07ae358459c997c90cc486cfbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57674
I20260620 14:26:17.105549 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet b710e07ae358459c997c90cc486cfbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54824
I20260620 14:26:17.105773 8448 tablet_replica.cc:330] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: stopping tablet replica
I20260620 14:26:17.106002 8450 tablet_replica.cc:330] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: stopping tablet replica
I20260620 14:26:17.106043 8448 raft_consensus.cc:2229] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:17.106313 8448 raft_consensus.cc:2258] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.106313 8450 raft_consensus.cc:2229] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:17.106492 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:26:17.106673 8450 raft_consensus.cc:2258] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:26:17.106869 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:26:17.107647 8448 ts_tablet_manager.cc:1905] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.107933 8450 ts_tablet_manager.cc:1905] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.110101 8448 ts_tablet_manager.cc:1918] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:17.110184 8448 log.cc:1196] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/b710e07ae358459c997c90cc486cfbf4
I20260620 14:26:17.110328 8450 ts_tablet_manager.cc:1918] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:17.110421 8450 log.cc:1196] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/b710e07ae358459c997c90cc486cfbf4
I20260620 14:26:17.110471 8448 ts_tablet_manager.cc:1939] T b710e07ae358459c997c90cc486cfbf4 P 78abda291c3e49089c19e4fa191cd82b: Deleting consensus metadata
I20260620 14:26:17.110729 8450 ts_tablet_manager.cc:1939] T b710e07ae358459c997c90cc486cfbf4 P 5e665f1796704b0aae9a3e2f08f507f6: Deleting consensus metadata
I20260620 14:26:17.111776 8449 tablet_replica.cc:330] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: stopping tablet replica
I20260620 14:26:17.111963 7900 catalog_manager.cc:4841] TS 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667): tablet b710e07ae358459c997c90cc486cfbf4 (table test-table [id=d84185d123784fc8880991e65ab36dc8]) successfully deleted
I20260620 14:26:17.112006 7899 catalog_manager.cc:4841] TS 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085): tablet b710e07ae358459c997c90cc486cfbf4 (table test-table [id=d84185d123784fc8880991e65ab36dc8]) successfully deleted
I20260620 14:26:17.112085 8051 tablet_service.cc:1467] Processing CreateTablet for tablet b81ede344b5e4694a7eeec53e45d9dd3 (DEFAULT_TABLE table=test-table [id=fb3e0259ad4c4e95b3144247870de9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:17.112319 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81ede344b5e4694a7eeec53e45d9dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.112490 8191 tablet_service.cc:1467] Processing CreateTablet for tablet b81ede344b5e4694a7eeec53e45d9dd3 (DEFAULT_TABLE table=test-table [id=fb3e0259ad4c4e95b3144247870de9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:17.112689 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81ede344b5e4694a7eeec53e45d9dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.112836 8331 tablet_service.cc:1467] Processing CreateTablet for tablet b81ede344b5e4694a7eeec53e45d9dd3 (DEFAULT_TABLE table=test-table [id=fb3e0259ad4c4e95b3144247870de9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:17.113051 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b81ede344b5e4694a7eeec53e45d9dd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.114248 8426 tablet_bootstrap.cc:492] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6: Bootstrap starting.
I20260620 14:26:17.114589 8449 raft_consensus.cc:2229] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:26:17.114877 8449 raft_consensus.cc:2258] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.115021 8426 tablet_bootstrap.cc:654] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.115307 8427 tablet_bootstrap.cc:492] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: Bootstrap starting.
I20260620 14:26:17.115386 8425 tablet_bootstrap.cc:492] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: Bootstrap starting.
I20260620 14:26:17.116151 8427 tablet_bootstrap.cc:654] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.116163 8425 tablet_bootstrap.cc:654] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.116474 8426 tablet_bootstrap.cc:492] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6: No bootstrap required, opened a new log
I20260620 14:26:17.116552 8426 ts_tablet_manager.cc:1397] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:17.117036 8426 raft_consensus.cc:348] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.117149 8426 raft_consensus.cc:374] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.117182 8426 raft_consensus.cc:729] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e665f1796704b0aae9a3e2f08f507f6, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.117316 8426 consensus_queue.cc:260] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.117481 8449 ts_tablet_manager.cc:1905] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.118960 8426 ts_tablet_manager.cc:1428] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:26:17.118994 8425 tablet_bootstrap.cc:492] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: No bootstrap required, opened a new log
I20260620 14:26:17.119068 8425 ts_tablet_manager.cc:1397] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:26:17.119457 8425 raft_consensus.cc:348] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.119553 8425 raft_consensus.cc:374] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.119585 8425 raft_consensus.cc:729] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78abda291c3e49089c19e4fa191cd82b, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.119650 8449 ts_tablet_manager.cc:1918] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:17.119721 8449 log.cc:1196] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/b710e07ae358459c997c90cc486cfbf4
I20260620 14:26:17.119707 8425 consensus_queue.cc:260] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.119976 8449 ts_tablet_manager.cc:1939] T b710e07ae358459c997c90cc486cfbf4 P 8b360b362a904967b76e36a73e9ecd32: Deleting consensus metadata
I20260620 14:26:17.119971 8425 ts_tablet_manager.cc:1428] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:17.120959 7898 catalog_manager.cc:4841] TS 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209): tablet b710e07ae358459c997c90cc486cfbf4 (table test-table [id=d84185d123784fc8880991e65ab36dc8]) successfully deleted
I20260620 14:26:17.121852 8427 tablet_bootstrap.cc:492] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: No bootstrap required, opened a new log
I20260620 14:26:17.121932 8427 ts_tablet_manager.cc:1397] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 14:26:17.122301 8427 raft_consensus.cc:348] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.122396 8427 raft_consensus.cc:374] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.122433 8427 raft_consensus.cc:729] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b360b362a904967b76e36a73e9ecd32, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.122536 8427 consensus_queue.cc:260] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.122767 8427 ts_tablet_manager.cc:1428] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260620 14:26:17.181320 8398 tablet.cc:2369] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:26:17.236495 8258 tablet.cc:2369] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:26:17.317654 8443 raft_consensus.cc:482] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:17.317762 8443 raft_consensus.cc:504] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.318148 8443 leader_election.cc:288] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667), 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085)
I20260620 14:26:17.318531 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81ede344b5e4694a7eeec53e45d9dd3" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78abda291c3e49089c19e4fa191cd82b" is_pre_election: true
I20260620 14:26:17.318543 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81ede344b5e4694a7eeec53e45d9dd3" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6" is_pre_election: true
I20260620 14:26:17.318693 8211 raft_consensus.cc:2454] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b360b362a904967b76e36a73e9ecd32 in term 0.
I20260620 14:26:17.318698 8071 raft_consensus.cc:2454] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b360b362a904967b76e36a73e9ecd32 in term 0.
I20260620 14:26:17.318981 8283 leader_election.cc:302] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78abda291c3e49089c19e4fa191cd82b, 8b360b362a904967b76e36a73e9ecd32; no voters:
I20260620 14:26:17.319164 8443 raft_consensus.cc:2781] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:17.319223 8443 raft_consensus.cc:482] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:17.319262 8443 raft_consensus.cc:3037] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.320186 8443 raft_consensus.cc:504] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.320520 8443 leader_election.cc:288] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 election: Requested vote from peers 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667), 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085)
I20260620 14:26:17.320828 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81ede344b5e4694a7eeec53e45d9dd3" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6"
I20260620 14:26:17.320832 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b81ede344b5e4694a7eeec53e45d9dd3" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78abda291c3e49089c19e4fa191cd82b"
I20260620 14:26:17.320945 8071 raft_consensus.cc:3037] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.320945 8211 raft_consensus.cc:3037] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.322031 8211 raft_consensus.cc:2454] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b360b362a904967b76e36a73e9ecd32 in term 1.
I20260620 14:26:17.322031 8071 raft_consensus.cc:2454] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b360b362a904967b76e36a73e9ecd32 in term 1.
I20260620 14:26:17.322309 8283 leader_election.cc:302] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78abda291c3e49089c19e4fa191cd82b, 8b360b362a904967b76e36a73e9ecd32; no voters:
I20260620 14:26:17.322506 8443 raft_consensus.cc:2781] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:17.322605 8443 raft_consensus.cc:686] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 1 LEADER]: Becoming Leader. State: Replica: 8b360b362a904967b76e36a73e9ecd32, State: Running, Role: LEADER
I20260620 14:26:17.322783 8443 consensus_queue.cc:237] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.324323 7913 catalog_manager.cc:5495] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8b360b362a904967b76e36a73e9ecd32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } health_report { overall_health: HEALTHY } } }
Jun 20 14:26:17 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965576, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:17.390282 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.382636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.0.98.62:45233)
0620 14:26:17.382753 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.382757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.382909 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:26:17.383024 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.383026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.383060 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.383121 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:17.383680 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.384285 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.384934 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.385115 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.385183 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.387277 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:17.387285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:17.387287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:17.387305 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:17.388673 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:17.389110 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:17.389113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:17.389115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:17.389181 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:17.389473 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:17.389478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:17.389480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:17.389632 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:17.389733 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.390007 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.390148 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":46}
I20260620 14:26:17.390295 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.382597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45233 (local address 127.0.0.1:45594)
0620 14:26:17.382833 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:17.382845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:17.382927 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:17.383151 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:17.383153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:17.383519 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:17.383598 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.383603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.384369 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.384371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.384851 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.384857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.385219 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.385222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.385276 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.385740 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:17.385750 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:17.387148 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:17.388792 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:17.388801 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:17.388807 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:17.389013 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:17.389283 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:17.389288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:17.389289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:17.389380 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:17.389733 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:17.389740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:17.389848 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 14:26:17.389965 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:26:17.390098 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260620 14:26:17.393602 7913 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45594:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:26:17.394178 7913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:26:17.398164 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 87dc575f57354eb897d52d330f38cbf0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59e47dcdf23243daa4f398f0fca674e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:26:17.398232 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 87dc575f57354eb897d52d330f38cbf0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59e47dcdf23243daa4f398f0fca674e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:26:17.398283 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 87dc575f57354eb897d52d330f38cbf0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59e47dcdf23243daa4f398f0fca674e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:26:17.398559 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87dc575f57354eb897d52d330f38cbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.398582 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87dc575f57354eb897d52d330f38cbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.398582 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87dc575f57354eb897d52d330f38cbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.400877 8426 tablet_bootstrap.cc:492] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: Bootstrap starting.
I20260620 14:26:17.401064 8427 tablet_bootstrap.cc:492] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: Bootstrap starting.
I20260620 14:26:17.401413 8425 tablet_bootstrap.cc:492] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: Bootstrap starting.
I20260620 14:26:17.401876 8426 tablet_bootstrap.cc:654] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.401913 8427 tablet_bootstrap.cc:654] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.402262 8425 tablet_bootstrap.cc:654] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.402985 8427 tablet_bootstrap.cc:492] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: No bootstrap required, opened a new log
I20260620 14:26:17.403074 8427 ts_tablet_manager.cc:1397] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:17.403385 8425 tablet_bootstrap.cc:492] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: No bootstrap required, opened a new log
I20260620 14:26:17.403439 8425 ts_tablet_manager.cc:1397] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:17.403515 8426 tablet_bootstrap.cc:492] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: No bootstrap required, opened a new log
I20260620 14:26:17.403498 8427 raft_consensus.cc:348] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.403581 8426 ts_tablet_manager.cc:1397] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:26:17.403602 8427 raft_consensus.cc:374] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.403635 8427 raft_consensus.cc:729] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b360b362a904967b76e36a73e9ecd32, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.403759 8427 consensus_queue.cc:260] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.403769 8425 raft_consensus.cc:348] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.403863 8425 raft_consensus.cc:374] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.403884 8425 raft_consensus.cc:729] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78abda291c3e49089c19e4fa191cd82b, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.403997 8427 ts_tablet_manager.cc:1428] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:26:17.403990 8426 raft_consensus.cc:348] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.403999 8425 consensus_queue.cc:260] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.404098 8426 raft_consensus.cc:374] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.404134 8426 raft_consensus.cc:729] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e665f1796704b0aae9a3e2f08f507f6, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.404278 8425 ts_tablet_manager.cc:1428] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:17.404263 8426 consensus_queue.cc:260] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.404512 8426 ts_tablet_manager.cc:1428] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:17.404459 8431 tablet_replica.cc:439] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } }
I20260620 14:26:17.404585 8431 tablet_replica.cc:442] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:26:17.404577 8432 tablet_replica.cc:439] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } }
I20260620 14:26:17.404707 8432 tablet_replica.cc:442] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:26:17.404721 8443 tablet_replica.cc:439] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } }
I20260620 14:26:17.404841 8443 tablet_replica.cc:442] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:26:17.429304 8431 raft_consensus.cc:482] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:17.429389 8431 raft_consensus.cc:504] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.430380 8431 leader_election.cc:288] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667), 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209)
I20260620 14:26:17.434536 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.430677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42919 (local address 127.0.98.3:33209)
0620 14:26:17.430788 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.430793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.430969 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:26:17.431157 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.431159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.431203 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.431274 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:17.431607 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.432355 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.433549 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.433967 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.434084 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.434165 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.434353 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.434424 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:26:17.434563 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.430608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41659 (local address 127.0.98.1:37667)
0620 14:26:17.430726 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.430731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.430917 (+ 186us) server_negotiation.cc:407] Connection header received
0620 14:26:17.431107 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.431113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.431173 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.431255 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:17.431604 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.432349 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.433606 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.434052 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.434150 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.434206 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.434387 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.434458 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:26:17.434561 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.430597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33209 (local address 127.0.98.2:42919)
0620 14:26:17.430872 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:17.430886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:17.431006 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:17.431299 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:17.431302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:17.431333 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:17.431486 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.431493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.432449 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.432452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.433435 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.433444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.434073 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.434079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.434225 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.434252 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:26:17.434317 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:26:17.434401 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":6,"thread_start_us":77,"threads_started":1}
I20260620 14:26:17.434573 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.430523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37667 (local address 127.0.98.2:41659)
0620 14:26:17.430817 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:17.430851 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:17.430992 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:17.431265 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:17.431268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:17.431282 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:17.431480 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.431488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.432494 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.432497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.433468 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.433476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.434153 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.434156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.434255 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.434271 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:26:17.434361 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:26:17.434432 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":76,"thread_start_us":67,"threads_started":1}
I20260620 14:26:17.435191 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87dc575f57354eb897d52d330f38cbf0" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6" is_pre_election: true
I20260620 14:26:17.435305 8071 raft_consensus.cc:2454] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 0.
I20260620 14:26:17.435261 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87dc575f57354eb897d52d330f38cbf0" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b360b362a904967b76e36a73e9ecd32" is_pre_election: true
I20260620 14:26:17.435487 8351 raft_consensus.cc:2454] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 0.
I20260620 14:26:17.435669 8144 leader_election.cc:302] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e665f1796704b0aae9a3e2f08f507f6, 78abda291c3e49089c19e4fa191cd82b; no voters:
I20260620 14:26:17.435899 8431 raft_consensus.cc:2781] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:17.435954 8431 raft_consensus.cc:482] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:17.435990 8431 raft_consensus.cc:3037] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.436887 8431 raft_consensus.cc:504] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.437233 8431 leader_election.cc:288] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 election: Requested vote from peers 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667), 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209)
I20260620 14:26:17.437510 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87dc575f57354eb897d52d330f38cbf0" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6"
I20260620 14:26:17.437547 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87dc575f57354eb897d52d330f38cbf0" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b360b362a904967b76e36a73e9ecd32"
I20260620 14:26:17.437623 8071 raft_consensus.cc:3037] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.437656 8351 raft_consensus.cc:3037] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.438397 8071 raft_consensus.cc:2454] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 1.
I20260620 14:26:17.438512 8351 raft_consensus.cc:2454] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 1.
I20260620 14:26:17.438638 8144 leader_election.cc:302] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e665f1796704b0aae9a3e2f08f507f6, 78abda291c3e49089c19e4fa191cd82b; no voters:
I20260620 14:26:17.438803 8431 raft_consensus.cc:2781] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:17.438973 8431 raft_consensus.cc:686] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [term 1 LEADER]: Becoming Leader. State: Replica: 78abda291c3e49089c19e4fa191cd82b, State: Running, Role: LEADER
I20260620 14:26:17.439201 8431 consensus_queue.cc:237] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.440475 8466 tablet_replica.cc:439] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: TxnStatusTablet state changed. Reason: New leader 78abda291c3e49089c19e4fa191cd82b. Latest consensus state: current_term: 1 leader_uuid: "78abda291c3e49089c19e4fa191cd82b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } }
I20260620 14:26:17.440579 8466 tablet_replica.cc:442] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:26:17.440814 8468 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:26:17.440951 8468 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:26:17.441504 7912 catalog_manager.cc:5495] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b reported cstate change: term changed from 0 to 1, leader changed from <none> to 78abda291c3e49089c19e4fa191cd82b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78abda291c3e49089c19e4fa191cd82b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } health_report { overall_health: UNKNOWN } } }
I20260620 14:26:17.458536 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.454723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37085 (local address 127.0.0.1:57928)
0620 14:26:17.454847 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:17.454885 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:17.455004 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:17.455366 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:17.455368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:17.455375 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:26:17.455475 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.455479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.456395 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.456397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.456983 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.456990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.457531 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.457534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.457604 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.457642 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:26:17.458319 (+ 677us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:26:17.458327 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:26:17.458375 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:26:17.458443 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:26:17.458570 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.454802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.0.98.2:37085)
0620 14:26:17.454929 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.454932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.454945 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:17.455157 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.455160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.455213 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.455323 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:26:17.455569 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.456306 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.457083 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.457435 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.457541 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.457763 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:26:17.458234 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:26:17.458337 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.458388 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.458435 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:26:17.460783 8071 raft_consensus.cc:1261] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Refusing update from remote peer 78abda291c3e49089c19e4fa191cd82b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:17.460953 8351 raft_consensus.cc:1261] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Refusing update from remote peer 78abda291c3e49089c19e4fa191cd82b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:17.461222 8466 consensus_queue.cc:1035] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:17.461419 8431 consensus_queue.cc:1035] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:17.462836 8443 tablet_replica.cc:439] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: TxnStatusTablet state changed. Reason: New leader 78abda291c3e49089c19e4fa191cd82b. Latest consensus state: current_term: 1 leader_uuid: "78abda291c3e49089c19e4fa191cd82b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } }
I20260620 14:26:17.462934 8443 tablet_replica.cc:442] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:26:17.463160 8432 tablet_replica.cc:439] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: TxnStatusTablet state changed. Reason: New leader 78abda291c3e49089c19e4fa191cd82b. Latest consensus state: current_term: 1 leader_uuid: "78abda291c3e49089c19e4fa191cd82b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } }
I20260620 14:26:17.463259 8432 tablet_replica.cc:442] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:26:17.464408 8466 tablet_replica.cc:439] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78abda291c3e49089c19e4fa191cd82b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } }
I20260620 14:26:17.464498 8466 tablet_replica.cc:442] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:26:17.466507 8431 tablet_replica.cc:439] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78abda291c3e49089c19e4fa191cd82b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } }
I20260620 14:26:17.466612 8431 tablet_replica.cc:442] T 87dc575f57354eb897d52d330f38cbf0 P 78abda291c3e49089c19e4fa191cd82b: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:26:17.466964 8432 tablet_replica.cc:439] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78abda291c3e49089c19e4fa191cd82b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } }
I20260620 14:26:17.467057 8432 tablet_replica.cc:442] T 87dc575f57354eb897d52d330f38cbf0 P 5e665f1796704b0aae9a3e2f08f507f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:26:17.467291 8438 mvcc.cc:205] Tried to move back new op lower bound from 7298931005275541504 to 7298931005193547776. Current Snapshot: MvccSnapshot[applied={T|T < 7298931005275541504 or (T in {7298931005275541504})}]
I20260620 14:26:17.467624 8443 tablet_replica.cc:439] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78abda291c3e49089c19e4fa191cd82b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } }
I20260620 14:26:17.467732 8443 tablet_replica.cc:442] T 87dc575f57354eb897d52d330f38cbf0 P 8b360b362a904967b76e36a73e9ecd32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:26:17.479342 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.476137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45267 (local address 127.0.98.2:37085)
0620 14:26:17.476226 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.476230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.476375 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:26:17.476458 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.476461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.476500 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.476579 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:26:17.476780 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.477539 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.478317 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.478639 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.478741 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.478962 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:26:17.479028 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:26:17.479095 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.479175 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.479234 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:26:17.479393 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.476060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37085 (local address 127.0.98.3:45267)
0620 14:26:17.476300 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:17.476312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:17.476385 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:17.476591 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:17.476593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:17.476598 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:26:17.476674 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.476677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.477632 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.477634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.478215 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.478222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.478743 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.478746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.478800 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.478835 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:26:17.479113 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:26:17.479117 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:26:17.479166 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:26:17.479221 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260620 14:26:17.485255 8211 raft_consensus.cc:1261] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Refusing update from remote peer 8b360b362a904967b76e36a73e9ecd32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:17.485340 8071 raft_consensus.cc:1261] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Refusing update from remote peer 8b360b362a904967b76e36a73e9ecd32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:17.485595 8443 consensus_queue.cc:1035] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:17.485710 8435 consensus_queue.cc:1035] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:17.506489 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.502529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33209 (local address 127.0.98.2:45085)
0620 14:26:17.502849 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:17.502863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:17.502960 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:17.503197 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:17.503200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:17.503206 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:26:17.503317 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.503322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.504291 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.504294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.505303 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.505313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.505740 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.505743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.505825 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.505868 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:26:17.506208 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:26:17.506214 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:26:17.506289 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:26:17.506352 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":102,"threads_started":1}
I20260620 14:26:17.506479 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.502664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45085 (local address 127.0.98.3:33209)
0620 14:26:17.502795 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.502799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.502938 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:26:17.503037 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.503039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.503074 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.503178 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:26:17.503433 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.504160 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.505407 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.505643 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.505716 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.506011 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:26:17.506078 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:26:17.506163 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.506301 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.506346 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:26:17.530521 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:17.530668 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:17.532500 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5: Sending DeleteTablet for 3 replicas of tablet b81ede344b5e4694a7eeec53e45d9dd3
I20260620 14:26:17.532932 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet b81ede344b5e4694a7eeec53e45d9dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57674
I20260620 14:26:17.533283 8450 tablet_replica.cc:330] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6: stopping tablet replica
I20260620 14:26:17.533280 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet b81ede344b5e4694a7eeec53e45d9dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57922
I20260620 14:26:17.533397 8450 raft_consensus.cc:2229] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:17.533427 8448 tablet_replica.cc:330] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: stopping tablet replica
I20260620 14:26:17.533486 8450 raft_consensus.cc:2258] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.533527 8448 raft_consensus.cc:2229] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:17.533607 8448 raft_consensus.cc:2258] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.533668 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet b81ede344b5e4694a7eeec53e45d9dd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54824
I20260620 14:26:17.533828 8449 tablet_replica.cc:330] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: stopping tablet replica
I20260620 14:26:17.533921 8449 raft_consensus.cc:2229] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:26:17.534071 8449 raft_consensus.cc:2258] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.534317 8448 ts_tablet_manager.cc:1905] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.534698 8449 ts_tablet_manager.cc:1905] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.534732 8450 ts_tablet_manager.cc:1905] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.536580 8448 ts_tablet_manager.cc:1918] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:26:17.536628 8448 log.cc:1196] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/b81ede344b5e4694a7eeec53e45d9dd3
I20260620 14:26:17.536805 8448 ts_tablet_manager.cc:1939] T b81ede344b5e4694a7eeec53e45d9dd3 P 78abda291c3e49089c19e4fa191cd82b: Deleting consensus metadata
I20260620 14:26:17.537168 8449 ts_tablet_manager.cc:1918] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:26:17.537252 8449 log.cc:1196] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/b81ede344b5e4694a7eeec53e45d9dd3
I20260620 14:26:17.537251 8450 ts_tablet_manager.cc:1918] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:26:17.537310 8450 log.cc:1196] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/b81ede344b5e4694a7eeec53e45d9dd3
I20260620 14:26:17.537367 7899 catalog_manager.cc:4841] TS 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085): tablet b81ede344b5e4694a7eeec53e45d9dd3 (table test-table [id=fb3e0259ad4c4e95b3144247870de9ab]) successfully deleted
I20260620 14:26:17.537510 8449 ts_tablet_manager.cc:1939] T b81ede344b5e4694a7eeec53e45d9dd3 P 8b360b362a904967b76e36a73e9ecd32: Deleting consensus metadata
I20260620 14:26:17.537602 8450 ts_tablet_manager.cc:1939] T b81ede344b5e4694a7eeec53e45d9dd3 P 5e665f1796704b0aae9a3e2f08f507f6: Deleting consensus metadata
I20260620 14:26:17.538110 7898 catalog_manager.cc:4841] TS 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209): tablet b81ede344b5e4694a7eeec53e45d9dd3 (table test-table [id=fb3e0259ad4c4e95b3144247870de9ab]) successfully deleted
I20260620 14:26:17.538129 7900 catalog_manager.cc:4841] TS 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667): tablet b81ede344b5e4694a7eeec53e45d9dd3 (table test-table [id=fb3e0259ad4c4e95b3144247870de9ab]) successfully deleted
Jun 20 14:26:17 dist-test-slave-pk3j krb5kdc[7869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:26:17 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:17 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:26:17.560860 7963 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:17.554774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.0.98.62:45233)
0620 14:26:17.554895 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.554900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.555048 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:26:17.555150 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.555153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.555198 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.555278 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:17.555851 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.556536 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.557427 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.557773 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.557875 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.560750 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45596: BlockingRecv error: recv got EOF from 127.0.0.1:45596 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 14:26:17 dist-test-slave-pk3j krb5kdc[7869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 14:26:17.570003 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.562145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.0.98.62:45233)
0620 14:26:17.562241 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.562244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.562381 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:26:17.562511 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.562514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.562552 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.562624 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:17.563114 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.563596 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.564273 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.564501 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.564597 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.567054 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:17.567057 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:17.567059 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:17.567079 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:17.568370 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:17.568798 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:17.568800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:17.568801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:17.568880 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:17.569122 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:17.569124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:17.569125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:17.569398 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:17.569489 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.569739 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.569816 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:26:17.572306 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:26:17.572747 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:26:17.577149 8051 tablet_service.cc:1467] Processing CreateTablet for tablet debb90070e7e43508fe1a6530cd161be (DEFAULT_TABLE table=test-table [id=1ea53f197539443a94aa475cebe1c287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:17.577471 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debb90070e7e43508fe1a6530cd161be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.577522 8191 tablet_service.cc:1467] Processing CreateTablet for tablet debb90070e7e43508fe1a6530cd161be (DEFAULT_TABLE table=test-table [id=1ea53f197539443a94aa475cebe1c287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:17.577754 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debb90070e7e43508fe1a6530cd161be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.579800 8426 tablet_bootstrap.cc:492] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6: Bootstrap starting.
I20260620 14:26:17.580144 8331 tablet_service.cc:1467] Processing CreateTablet for tablet debb90070e7e43508fe1a6530cd161be (DEFAULT_TABLE table=test-table [id=1ea53f197539443a94aa475cebe1c287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:17.580435 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debb90070e7e43508fe1a6530cd161be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.580709 8426 tablet_bootstrap.cc:654] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.582192 8426 tablet_bootstrap.cc:492] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6: No bootstrap required, opened a new log
I20260620 14:26:17.582286 8426 ts_tablet_manager.cc:1397] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:26:17.582472 8427 tablet_bootstrap.cc:492] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32: Bootstrap starting.
I20260620 14:26:17.582572 8425 tablet_bootstrap.cc:492] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b: Bootstrap starting.
I20260620 14:26:17.582747 8426 raft_consensus.cc:348] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.582854 8426 raft_consensus.cc:374] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.582886 8426 raft_consensus.cc:729] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e665f1796704b0aae9a3e2f08f507f6, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.583035 8426 consensus_queue.cc:260] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.583267 8427 tablet_bootstrap.cc:654] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.583376 8425 tablet_bootstrap.cc:654] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.583490 8426 ts_tablet_manager.cc:1428] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:26:17.584618 8425 tablet_bootstrap.cc:492] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b: No bootstrap required, opened a new log
I20260620 14:26:17.584703 8425 ts_tablet_manager.cc:1397] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:17.585130 8425 raft_consensus.cc:348] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.585274 8425 raft_consensus.cc:374] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.585325 8425 raft_consensus.cc:729] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78abda291c3e49089c19e4fa191cd82b, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.585453 8425 consensus_queue.cc:260] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.585729 8425 ts_tablet_manager.cc:1428] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:17.586773 8427 tablet_bootstrap.cc:492] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32: No bootstrap required, opened a new log
I20260620 14:26:17.586861 8427 ts_tablet_manager.cc:1397] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:26:17.587284 8427 raft_consensus.cc:348] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.587368 8427 raft_consensus.cc:374] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.587389 8427 raft_consensus.cc:729] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b360b362a904967b76e36a73e9ecd32, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.587503 8427 consensus_queue.cc:260] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.587749 8427 ts_tablet_manager.cc:1428] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:17.603783 8466 raft_consensus.cc:482] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:17.603901 8466 raft_consensus.cc:504] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.604245 8433 raft_consensus.cc:482] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:17.604290 8466 leader_election.cc:288] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667), 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209)
I20260620 14:26:17.604323 8433 raft_consensus.cc:504] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.604645 8433 leader_election.cc:288] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667), 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085)
I20260620 14:26:17.604755 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb90070e7e43508fe1a6530cd161be" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6" is_pre_election: true
I20260620 14:26:17.604817 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb90070e7e43508fe1a6530cd161be" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b360b362a904967b76e36a73e9ecd32" is_pre_election: true
I20260620 14:26:17.604861 8070 raft_consensus.cc:2454] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 0.
I20260620 14:26:17.604923 8351 raft_consensus.cc:2454] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 0.
I20260620 14:26:17.605239 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb90070e7e43508fe1a6530cd161be" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78abda291c3e49089c19e4fa191cd82b" is_pre_election: true
I20260620 14:26:17.605257 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb90070e7e43508fe1a6530cd161be" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6" is_pre_election: true
I20260620 14:26:17.605360 8070 raft_consensus.cc:2454] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b360b362a904967b76e36a73e9ecd32 in term 0.
I20260620 14:26:17.605333 8144 leader_election.cc:302] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e665f1796704b0aae9a3e2f08f507f6, 78abda291c3e49089c19e4fa191cd82b; no voters:
I20260620 14:26:17.605399 8211 raft_consensus.cc:2454] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b360b362a904967b76e36a73e9ecd32 in term 0.
I20260620 14:26:17.605528 8466 raft_consensus.cc:2781] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:17.605587 8466 raft_consensus.cc:482] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:17.605630 8466 raft_consensus.cc:3037] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.605695 8284 leader_election.cc:302] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e665f1796704b0aae9a3e2f08f507f6, 8b360b362a904967b76e36a73e9ecd32; no voters:
I20260620 14:26:17.605840 8433 raft_consensus.cc:2781] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:17.605886 8433 raft_consensus.cc:482] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:17.605933 8433 raft_consensus.cc:3037] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.606678 8466 raft_consensus.cc:504] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.606951 8433 raft_consensus.cc:504] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.607039 8466 leader_election.cc:288] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 election: Requested vote from peers 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667), 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209)
I20260620 14:26:17.607254 8433 leader_election.cc:288] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 election: Requested vote from peers 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667), 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085)
I20260620 14:26:17.607378 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb90070e7e43508fe1a6530cd161be" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b360b362a904967b76e36a73e9ecd32"
I20260620 14:26:17.607378 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb90070e7e43508fe1a6530cd161be" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6"
I20260620 14:26:17.607472 8070 raft_consensus.cc:3037] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.607517 8351 raft_consensus.cc:2379] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 78abda291c3e49089c19e4fa191cd82b in current term 1: Already voted for candidate 8b360b362a904967b76e36a73e9ecd32 in this term.
I20260620 14:26:17.607630 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb90070e7e43508fe1a6530cd161be" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78abda291c3e49089c19e4fa191cd82b"
I20260620 14:26:17.607770 8211 raft_consensus.cc:2379] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8b360b362a904967b76e36a73e9ecd32 in current term 1: Already voted for candidate 78abda291c3e49089c19e4fa191cd82b in this term.
I20260620 14:26:17.607955 8071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb90070e7e43508fe1a6530cd161be" candidate_uuid: "8b360b362a904967b76e36a73e9ecd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6"
I20260620 14:26:17.608505 8070 raft_consensus.cc:2454] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 1.
I20260620 14:26:17.608505 8283 leader_election.cc:302] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b360b362a904967b76e36a73e9ecd32; no voters: 5e665f1796704b0aae9a3e2f08f507f6, 78abda291c3e49089c19e4fa191cd82b
I20260620 14:26:17.608703 8433 raft_consensus.cc:2726] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 14:26:17.608798 8144 leader_election.cc:302] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e665f1796704b0aae9a3e2f08f507f6, 78abda291c3e49089c19e4fa191cd82b; no voters: 8b360b362a904967b76e36a73e9ecd32
I20260620 14:26:17.608975 8466 raft_consensus.cc:2781] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:17.609061 8466 raft_consensus.cc:686] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 1 LEADER]: Becoming Leader. State: Replica: 78abda291c3e49089c19e4fa191cd82b, State: Running, Role: LEADER
I20260620 14:26:17.609177 8466 consensus_queue.cc:237] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.610648 7913 catalog_manager.cc:5495] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b reported cstate change: term changed from 0 to 1, leader changed from <none> to 78abda291c3e49089c19e4fa191cd82b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78abda291c3e49089c19e4fa191cd82b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } health_report { overall_health: UNKNOWN } } }
I20260620 14:26:17.625015 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.622431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.0.98.2:37085)
0620 14:26:17.622550 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.622554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.622585 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:26:17.622693 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.622695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.622730 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.622782 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:26:17.622947 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.623407 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.624190 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.624378 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.624448 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.624643 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:26:17.624698 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:26:17.624747 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.624818 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.624851 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:26:17.626691 8351 raft_consensus.cc:1261] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Refusing update from remote peer 78abda291c3e49089c19e4fa191cd82b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:17.626735 8070 raft_consensus.cc:1261] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Refusing update from remote peer 78abda291c3e49089c19e4fa191cd82b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:17.627141 8466 consensus_queue.cc:1035] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:17.627290 8476 consensus_queue.cc:1035] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:17.635450 7914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:17.635566 7914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:17.637283 7914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5: Sending DeleteTablet for 3 replicas of tablet debb90070e7e43508fe1a6530cd161be
I20260620 14:26:17.637939 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet debb90070e7e43508fe1a6530cd161be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57674
I20260620 14:26:17.638015 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet debb90070e7e43508fe1a6530cd161be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57922
I20260620 14:26:17.638110 8450 tablet_replica.cc:330] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6: stopping tablet replica
I20260620 14:26:17.638191 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet debb90070e7e43508fe1a6530cd161be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54824
I20260620 14:26:17.638208 8450 raft_consensus.cc:2229] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:17.638345 8448 tablet_replica.cc:330] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b: stopping tablet replica
I20260620 14:26:17.638343 8450 raft_consensus.cc:2258] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.638361 8449 tablet_replica.cc:330] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32: stopping tablet replica
I20260620 14:26:17.638438 8448 raft_consensus.cc:2229] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:26:17.638479 8449 raft_consensus.cc:2229] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:17.638593 8449 raft_consensus.cc:2258] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.638646 8448 raft_consensus.cc:2258] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.639169 8450 ts_tablet_manager.cc:1905] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.639321 8448 ts_tablet_manager.cc:1905] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.639514 7914 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:26:17.639933 7914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:26:17.640897 8449 ts_tablet_manager.cc:1905] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.641412 8448 ts_tablet_manager.cc:1918] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:17.641407 8450 ts_tablet_manager.cc:1918] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:17.641471 8448 log.cc:1196] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/debb90070e7e43508fe1a6530cd161be
I20260620 14:26:17.641471 8450 log.cc:1196] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/debb90070e7e43508fe1a6530cd161be
I20260620 14:26:17.641717 8448 ts_tablet_manager.cc:1939] T debb90070e7e43508fe1a6530cd161be P 78abda291c3e49089c19e4fa191cd82b: Deleting consensus metadata
I20260620 14:26:17.641736 8450 ts_tablet_manager.cc:1939] T debb90070e7e43508fe1a6530cd161be P 5e665f1796704b0aae9a3e2f08f507f6: Deleting consensus metadata
I20260620 14:26:17.642221 7899 catalog_manager.cc:4841] TS 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085): tablet debb90070e7e43508fe1a6530cd161be (table test-table [id=1ea53f197539443a94aa475cebe1c287]) successfully deleted
I20260620 14:26:17.642633 7900 catalog_manager.cc:4841] TS 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667): tablet debb90070e7e43508fe1a6530cd161be (table test-table [id=1ea53f197539443a94aa475cebe1c287]) successfully deleted
I20260620 14:26:17.643038 8449 ts_tablet_manager.cc:1918] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:17.643121 8449 log.cc:1196] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/debb90070e7e43508fe1a6530cd161be
I20260620 14:26:17.643337 8449 ts_tablet_manager.cc:1939] T debb90070e7e43508fe1a6530cd161be P 8b360b362a904967b76e36a73e9ecd32: Deleting consensus metadata
I20260620 14:26:17.643949 7898 catalog_manager.cc:4841] TS 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209): tablet debb90070e7e43508fe1a6530cd161be (table test-table [id=1ea53f197539443a94aa475cebe1c287]) successfully deleted
I20260620 14:26:17.645481 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7d36a77c7943babe8e90e9ce50bd10 (DEFAULT_TABLE table=test-table [id=19b1a1c4cf554ca982bc3d5db9fcffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:17.645591 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7d36a77c7943babe8e90e9ce50bd10 (DEFAULT_TABLE table=test-table [id=19b1a1c4cf554ca982bc3d5db9fcffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:17.645697 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7d36a77c7943babe8e90e9ce50bd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.645807 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7d36a77c7943babe8e90e9ce50bd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.647440 8425 tablet_bootstrap.cc:492] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b: Bootstrap starting.
I20260620 14:26:17.647651 8426 tablet_bootstrap.cc:492] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6: Bootstrap starting.
I20260620 14:26:17.648144 8425 tablet_bootstrap.cc:654] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.648455 8426 tablet_bootstrap.cc:654] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.649166 8425 tablet_bootstrap.cc:492] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b: No bootstrap required, opened a new log
I20260620 14:26:17.649258 8425 ts_tablet_manager.cc:1397] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 14:26:17.649601 8425 raft_consensus.cc:348] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.649601 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7d36a77c7943babe8e90e9ce50bd10 (DEFAULT_TABLE table=test-table [id=19b1a1c4cf554ca982bc3d5db9fcffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:17.649695 8425 raft_consensus.cc:374] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.649730 8425 raft_consensus.cc:729] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78abda291c3e49089c19e4fa191cd82b, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.649804 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7d36a77c7943babe8e90e9ce50bd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:17.649849 8425 consensus_queue.cc:260] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.650071 8425 ts_tablet_manager.cc:1428] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:26:17.651623 8427 tablet_bootstrap.cc:492] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32: Bootstrap starting.
I20260620 14:26:17.652005 8426 tablet_bootstrap.cc:492] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6: No bootstrap required, opened a new log
I20260620 14:26:17.652084 8426 ts_tablet_manager.cc:1397] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 14:26:17.652311 8427 tablet_bootstrap.cc:654] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:17.652451 8426 raft_consensus.cc:348] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.652568 8426 raft_consensus.cc:374] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.652597 8426 raft_consensus.cc:729] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e665f1796704b0aae9a3e2f08f507f6, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.652702 8426 consensus_queue.cc:260] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.652942 8426 ts_tablet_manager.cc:1428] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 14:26:17.653258 8427 tablet_bootstrap.cc:492] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32: No bootstrap required, opened a new log
I20260620 14:26:17.653339 8427 ts_tablet_manager.cc:1397] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:26:17.653728 8427 raft_consensus.cc:348] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.653856 8427 raft_consensus.cc:374] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:17.653908 8427 raft_consensus.cc:729] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b360b362a904967b76e36a73e9ecd32, State: Initialized, Role: FOLLOWER
I20260620 14:26:17.654028 8427 consensus_queue.cc:260] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.654277 8427 ts_tablet_manager.cc:1428] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:26:17.699270 8476 raft_consensus.cc:482] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:17.699405 8476 raft_consensus.cc:504] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.699800 8476 leader_election.cc:288] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667), 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209)
I20260620 14:26:17.700111 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7d36a77c7943babe8e90e9ce50bd10" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6" is_pre_election: true
I20260620 14:26:17.700140 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7d36a77c7943babe8e90e9ce50bd10" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b360b362a904967b76e36a73e9ecd32" is_pre_election: true
I20260620 14:26:17.700244 8070 raft_consensus.cc:2454] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 0.
I20260620 14:26:17.700270 8351 raft_consensus.cc:2454] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 0.
I20260620 14:26:17.700519 8144 leader_election.cc:302] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e665f1796704b0aae9a3e2f08f507f6, 78abda291c3e49089c19e4fa191cd82b; no voters:
I20260620 14:26:17.700713 8476 raft_consensus.cc:2781] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:17.700793 8476 raft_consensus.cc:482] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:17.700838 8476 raft_consensus.cc:3037] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.701992 8476 raft_consensus.cc:504] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.702342 8476 leader_election.cc:288] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 election: Requested vote from peers 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667), 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209)
I20260620 14:26:17.702687 8070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7d36a77c7943babe8e90e9ce50bd10" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e665f1796704b0aae9a3e2f08f507f6"
I20260620 14:26:17.702698 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7d36a77c7943babe8e90e9ce50bd10" candidate_uuid: "78abda291c3e49089c19e4fa191cd82b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b360b362a904967b76e36a73e9ecd32"
I20260620 14:26:17.702817 8070 raft_consensus.cc:3037] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.702826 8351 raft_consensus.cc:3037] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:17.703583 8351 raft_consensus.cc:2454] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 1.
I20260620 14:26:17.703826 8070 raft_consensus.cc:2454] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78abda291c3e49089c19e4fa191cd82b in term 1.
I20260620 14:26:17.703827 8142 leader_election.cc:302] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78abda291c3e49089c19e4fa191cd82b, 8b360b362a904967b76e36a73e9ecd32; no voters:
I20260620 14:26:17.703958 8476 raft_consensus.cc:2781] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:17.704048 8476 raft_consensus.cc:686] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 1 LEADER]: Becoming Leader. State: Replica: 78abda291c3e49089c19e4fa191cd82b, State: Running, Role: LEADER
I20260620 14:26:17.704174 8476 consensus_queue.cc:237] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } }
I20260620 14:26:17.705780 7913 catalog_manager.cc:5495] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b reported cstate change: term changed from 0 to 1, leader changed from <none> to 78abda291c3e49089c19e4fa191cd82b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78abda291c3e49089c19e4fa191cd82b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78abda291c3e49089c19e4fa191cd82b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } health_report { overall_health: UNKNOWN } } }
I20260620 14:26:17.729532 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.726272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37085 (local address 127.0.98.2:35741)
0620 14:26:17.726400 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:17.726412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:17.726480 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:17.726679 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:17.726681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:17.726686 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:26:17.726783 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.726786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.727768 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.727771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.728458 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:17.728466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.728977 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.728979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:17.729031 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.729052 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:26:17.729296 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:26:17.729301 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:26:17.729366 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:26:17.729422 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:26:17.729538 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.726310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35741 (local address 127.0.98.2:37085)
0620 14:26:17.726410 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:17.726415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:17.726451 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:26:17.726544 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:17.726547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:17.726589 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:17.726666 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:26:17.726866 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.727678 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.728570 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:17.728880 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:17.728982 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:17.729138 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:26:17.729226 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:26:17.729287 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:17.729375 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 14:26:17.729418 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:26:17.734284 8351 raft_consensus.cc:1261] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Refusing update from remote peer 78abda291c3e49089c19e4fa191cd82b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:17.734315 8070 raft_consensus.cc:1261] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Refusing update from remote peer 78abda291c3e49089c19e4fa191cd82b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:17.734701 8476 consensus_queue.cc:1035] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e665f1796704b0aae9a3e2f08f507f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:17.734815 8431 consensus_queue.cc:1035] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b360b362a904967b76e36a73e9ecd32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:17.774016 7913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:17.774166 7913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:17.775875 7913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6906fc9a173349bfa33e2f6ea3ff7ce5: Sending DeleteTablet for 3 replicas of tablet 5c7d36a77c7943babe8e90e9ce50bd10
I20260620 14:26:17.776355 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7d36a77c7943babe8e90e9ce50bd10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57674
I20260620 14:26:17.776510 8450 tablet_replica.cc:330] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6: stopping tablet replica
I20260620 14:26:17.776500 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7d36a77c7943babe8e90e9ce50bd10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54824
I20260620 14:26:17.776607 8450 raft_consensus.cc:2229] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:17.776679 8449 tablet_replica.cc:330] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32: stopping tablet replica
I20260620 14:26:17.776746 8450 raft_consensus.cc:2258] T 5c7d36a77c7943babe8e90e9ce50bd10 P 5e665f1796704b0aae9a3e2f08f507f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.776798 8449 raft_consensus.cc:2229] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:17.776880 8449 raft_consensus.cc:2258] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.777124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 7986
I20260620 14:26:17.779116 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7d36a77c7943babe8e90e9ce50bd10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:17 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57922
I20260620 14:26:17.779302 8448 tablet_replica.cc:330] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b: stopping tablet replica
I20260620 14:26:17.779407 8448 raft_consensus.cc:2229] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:26:17.779584 8449 ts_tablet_manager.cc:1905] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.779600 8448 raft_consensus.cc:2258] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:17.780143 8448 ts_tablet_manager.cc:1905] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:17.781852 8449 ts_tablet_manager.cc:1918] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:26:17.781926 8449 log.cc:1196] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/5c7d36a77c7943babe8e90e9ce50bd10
I20260620 14:26:17.782141 8449 ts_tablet_manager.cc:1939] T 5c7d36a77c7943babe8e90e9ce50bd10 P 8b360b362a904967b76e36a73e9ecd32: Deleting consensus metadata
I20260620 14:26:17.782615 8448 ts_tablet_manager.cc:1918] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:26:17.782680 8448 log.cc:1196] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/5c7d36a77c7943babe8e90e9ce50bd10
I20260620 14:26:17.782697 7898 catalog_manager.cc:4841] TS 8b360b362a904967b76e36a73e9ecd32 (127.0.98.3:33209): tablet 5c7d36a77c7943babe8e90e9ce50bd10 (table test-table [id=19b1a1c4cf554ca982bc3d5db9fcffc7]) successfully deleted
I20260620 14:26:17.782941 8448 ts_tablet_manager.cc:1939] T 5c7d36a77c7943babe8e90e9ce50bd10 P 78abda291c3e49089c19e4fa191cd82b: Deleting consensus metadata
I20260620 14:26:17.783586 7899 catalog_manager.cc:4841] TS 78abda291c3e49089c19e4fa191cd82b (127.0.98.2:37085): tablet 5c7d36a77c7943babe8e90e9ce50bd10 (table test-table [id=19b1a1c4cf554ca982bc3d5db9fcffc7]) successfully deleted
W20260620 14:26:17.786190 7900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37667 (error 108)
I20260620 14:26:17.786322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 8126
I20260620 14:26:17.787737 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:17.787393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37667 (local address 127.0.0.1:57678)
0620 14:26:17.787608 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:17.787670 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
W20260620 14:26:17.787856 7900 catalog_manager.cc:4584] TS 5e665f1796704b0aae9a3e2f08f507f6 (127.0.98.1:37667): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c7d36a77c7943babe8e90e9ce50bd10: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37667: connect: Connection refused (error 111)
I20260620 14:26:17.793638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 8266
I20260620 14:26:17.799875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 7882
2026-06-20T14:26:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2815 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:26:17 dist-test-slave-pk3j krb5kdc[8513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:26:17 dist-test-slave-pk3j krb5kdc[8513](info): set up 2 sockets
Jun 20 14:26:17 dist-test-slave-pk3j krb5kdc[8513](info): commencing operation
krb5kdc: starting...
W20260620 14:26:18.797576 392 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:26:18 dist-test-slave-pk3j krb5kdc[8513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:26:18.903649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44589
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:19.004359 8526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:19.004563 8526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:19.004599 8526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:19.008455 8526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:19.008500 8526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:19.008518 8526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:19.008534 8526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:19.008549 8526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:19.013397 8526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:19.014175 8526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:19.015009 8526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:19.022519 8532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:19.022542 8531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:19.022548 8534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:19.022735 8526 server_base.cc:1034] running on GCE node
I20260620 14:26:19.023272 8526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:19.024255 8526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:19.025435 8526 hybrid_clock.cc:648] HybridClock initialized: now 1781965579025424 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:26:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1229 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:26:19 dist-test-slave-pk3j krb5kdc[8539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:26:19 dist-test-slave-pk3j krb5kdc[8539](info): set up 2 sockets
Jun 20 14:26:19 dist-test-slave-pk3j krb5kdc[8539](info): commencing operation
krb5kdc: starting...
W20260620 14:26:20.041790 392 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:26:20 dist-test-slave-pk3j krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:26:20.147619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41215
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:20.247598 8552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:20.247812 8552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:20.247848 8552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:20.251665 8552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:20.251708 8552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:20.251727 8552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:20.251776 8552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:20.251811 8552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:20.256699 8552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:20.257565 8552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:20.258438 8552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:20.266165 8557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:20.266170 8560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:20.266177 8558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:20.266575 8552 server_base.cc:1034] running on GCE node
I20260620 14:26:20.267158 8552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:20.268162 8552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:20.269325 8552 hybrid_clock.cc:648] HybridClock initialized: now 1781965580269297 us; error 40 us; skew 500 ppm
Jun 20 14:26:20 dist-test-slave-pk3j krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:20.278422 8552 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:26:20.279132 8552 webserver.cc:460] Webserver started at http://127.0.98.62:33801/ using document root <none> and password file <none>
I20260620 14:26:20.279577 8552 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:20.279632 8552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:20.279784 8552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:20.281553 8552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "260cd2b38b144a3196fa4fb283e0a3fc"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "845860e855a165f4446d304ba044bae7"
server_key_iv: "4092237fe8a5c8648ec231045061a21f"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.282017 8552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "260cd2b38b144a3196fa4fb283e0a3fc"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "845860e855a165f4446d304ba044bae7"
server_key_iv: "4092237fe8a5c8648ec231045061a21f"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.285125 8552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:26:20.287364 8567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:20.288443 8552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:20.288576 8552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "260cd2b38b144a3196fa4fb283e0a3fc"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "845860e855a165f4446d304ba044bae7"
server_key_iv: "4092237fe8a5c8648ec231045061a21f"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.288667 8552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:20.304589 8552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:20.306986 8552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:20.307163 8552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:20.313356 8552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41215
I20260620 14:26:20.313375 8619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41215 every 8 connection(s)
I20260620 14:26:20.314133 8552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:20.316474 8620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:20.321236 8620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc: Bootstrap starting.
I20260620 14:26:20.322396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 8552
I20260620 14:26:20.322489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:20.322744 392 external_mini_cluster.cc:1428] Setting key ae724ac27f8b4fde6e471a618a6e90cd
I20260620 14:26:20.323503 8620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:20.324246 8620 log.cc:824] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:20.326220 8620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc: No bootstrap required, opened a new log
I20260620 14:26:20.330163 8620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } }
I20260620 14:26:20.330415 8620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:20.330497 8620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 260cd2b38b144a3196fa4fb283e0a3fc, State: Initialized, Role: FOLLOWER
I20260620 14:26:20.330885 8620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } }
I20260620 14:26:20.330996 8620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:20.331038 8620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:20.331099 8620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:26:20 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:20.332034 8620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } }
I20260620 14:26:20.332331 8620 leader_election.cc:302] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 260cd2b38b144a3196fa4fb283e0a3fc; no voters:
I20260620 14:26:20.332639 8620 leader_election.cc:288] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:20.332701 8625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:20.332899 8625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 1 LEADER]: Becoming Leader. State: Replica: 260cd2b38b144a3196fa4fb283e0a3fc, State: Running, Role: LEADER
I20260620 14:26:20.333236 8625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } }
I20260620 14:26:20.333788 8620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:20.334741 8626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } } }
I20260620 14:26:20.334849 8626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:20.335202 8634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:20.335166 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 260cd2b38b144a3196fa4fb283e0a3fc. Latest consensus state: current_term: 1 leader_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } } }
I20260620 14:26:20.335355 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:20.337059 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:20.323703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.0.98.62:41215)
0620 14:26:20.324050 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:20.324064 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:20.324094 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:26:20.324696 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:20.324712 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:20.325009 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:20.325320 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:20.326128 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.327062 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.328245 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.328951 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.329069 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:20.331821 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:20.331826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:20.331836 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:20.331859 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:20.333628 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:20.334092 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:20.334099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:20.334104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:20.334229 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:20.334566 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:20.334571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:20.334574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:20.335040 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:20.335779 (+ 739us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:20.335925 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:26:20.336199 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":119,"threads_started":1}
I20260620 14:26:20.338953 8634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:20.344506 8634 catalog_manager.cc:1347] Generated new cluster ID: 2b3a9f74a74348fa90553f2535cfa6dc
I20260620 14:26:20.344563 8634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:20.360212 8634 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:20.361094 8634 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:20.367022 8634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc: Generated new TSK 0
I20260620 14:26:20.367733 8634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:26:20.392299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41215
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:20.493788 8646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:20.493991 8646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:20.494056 8646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:20.497931 8646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:20.497974 8646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:20.498057 8646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:20.502826 8646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:20.503651 8646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:20.504500 8646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:20.513151 8654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:20.513135 8652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:20.513353 8646 server_base.cc:1034] running on GCE node
W20260620 14:26:20.513159 8651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:20.513813 8646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:20.514379 8646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:20.515552 8646 hybrid_clock.cc:648] HybridClock initialized: now 1781965580515529 us; error 37 us; skew 500 ppm
Jun 20 14:26:20 dist-test-slave-pk3j krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:20.518128 8646 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:26:20.518900 8646 webserver.cc:460] Webserver started at http://127.0.98.1:42801/ using document root <none> and password file <none>
I20260620 14:26:20.519394 8646 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:20.519479 8646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:20.519681 8646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:20.521500 8646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "6c7491a183e3462d87dc306e14e6183b"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "18799280a1a99e3af70a4b5a257e2494"
server_key_iv: "24fadb8d8f2d515062ecbbbddc0efc78"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.522003 8646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6c7491a183e3462d87dc306e14e6183b"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "18799280a1a99e3af70a4b5a257e2494"
server_key_iv: "24fadb8d8f2d515062ecbbbddc0efc78"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.525326 8646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:26:20.527464 8661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:20.528504 8646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:20.528628 8646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "6c7491a183e3462d87dc306e14e6183b"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "18799280a1a99e3af70a4b5a257e2494"
server_key_iv: "24fadb8d8f2d515062ecbbbddc0efc78"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.528726 8646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:20.538378 8646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:20.540693 8646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:20.540867 8646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:20.541414 8646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:20.542096 8646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:20.542162 8646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:20.542227 8646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:20.542275 8646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:20.550333 8646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42263
I20260620 14:26:20.550413 8774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42263 every 8 connection(s)
I20260620 14:26:20.551074 8646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:26:20.557301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 8646
I20260620 14:26:20.557430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:26:20.557731 392 external_mini_cluster.cc:1428] Setting key 3253b8aa8b83b410dd2061700f540ebe
Jun 20 14:26:20 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:20.563668 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:20.552665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47271 (local address 127.0.98.62:41215)
0620 14:26:20.552794 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:20.552799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:20.553549 (+ 750us) server_negotiation.cc:407] Connection header received
0620 14:26:20.554708 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:20.554710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:20.554753 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:20.554822 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:20.555764 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.556363 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.557681 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.557954 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.558037 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:20.560402 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:20.560409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:20.560415 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:20.560448 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:20.561798 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:20.562277 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:20.562282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:20.562283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:20.562362 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:20.562684 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:20.562689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:20.562691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:20.562854 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:20.562957 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:20.563390 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 14:26:20.563514 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:26:20.564344 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:20.553007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:47271)
0620 14:26:20.553424 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:20.553459 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:20.554539 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:20.554958 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:20.554964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:20.555251 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:20.555649 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:20.555660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.556469 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.556472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:20.557567 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:20.557579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.558099 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.558118 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:20.558261 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:20.558797 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:20.558811 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:20.560251 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:20.561898 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:20.561902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:20.561911 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:20.562174 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:20.562461 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:20.562465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:20.562467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:20.562591 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:20.562961 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:20.562969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:20.563171 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 14:26:20.563337 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 14:26:20.563605 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260620 14:26:20.565474 8775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41215
I20260620 14:26:20.565697 8775 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:20.566190 8775 heartbeater.cc:511] Master 127.0.98.62:41215 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:26:20.567696 8584 ts_manager.cc:194] Registered new tserver with Master: 6c7491a183e3462d87dc306e14e6183b (127.0.98.1:42263)
I20260620 14:26:20.568719 8584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:26:20.581583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41215
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:20.679903 8780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:20.680135 8780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:20.680205 8780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:20.683845 8780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:20.683918 8780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:20.684028 8780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:20.688759 8780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:20.689579 8780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:20.690404 8780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:20.698915 8786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:20.698928 8788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:20.698928 8785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:20.698997 8780 server_base.cc:1034] running on GCE node
I20260620 14:26:20.699481 8780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:20.700049 8780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:20.701252 8780 hybrid_clock.cc:648] HybridClock initialized: now 1781965580701228 us; error 42 us; skew 500 ppm
Jun 20 14:26:20 dist-test-slave-pk3j krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:20.703599 8780 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:26:20.704315 8780 webserver.cc:460] Webserver started at http://127.0.98.2:46239/ using document root <none> and password file <none>
I20260620 14:26:20.704804 8780 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:20.704886 8780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:20.705088 8780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:20.706864 8780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "a089986493ab44be9a877b64fc031eef"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "c6cf1b3e92b2fc8157a233a91a66fbb7"
server_key_iv: "3a12bdb0fb8158ce956d89fcf3b6f6c2"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.707393 8780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a089986493ab44be9a877b64fc031eef"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "c6cf1b3e92b2fc8157a233a91a66fbb7"
server_key_iv: "3a12bdb0fb8158ce956d89fcf3b6f6c2"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.710700 8780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:20.712725 8795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:20.713810 8780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:20.713935 8780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "a089986493ab44be9a877b64fc031eef"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "c6cf1b3e92b2fc8157a233a91a66fbb7"
server_key_iv: "3a12bdb0fb8158ce956d89fcf3b6f6c2"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.714030 8780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:20.725347 8780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:20.727783 8780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:20.727960 8780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:20.728459 8780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:20.729141 8780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:20.729206 8780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:20.729306 8780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:20.729357 8780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:20.737669 8780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40681
I20260620 14:26:20.737694 8908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40681 every 8 connection(s)
I20260620 14:26:20.738409 8780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:20.746482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 8780
I20260620 14:26:20.746621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
Jun 20 14:26:20 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:20.746867 392 external_mini_cluster.cc:1428] Setting key ece53114b898d6ab7d881983304cd19d
I20260620 14:26:20.751000 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:20.739886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52949 (local address 127.0.98.62:41215)
0620 14:26:20.740012 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:20.740016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:20.740656 (+ 640us) server_negotiation.cc:407] Connection header received
0620 14:26:20.741992 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:20.741995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:20.742052 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:20.742155 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:20.743137 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.743643 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.744692 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.744911 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.745016 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:20.747261 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:20.747266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:20.747274 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:20.747294 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:20.748979 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:20.749499 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:20.749506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:20.749511 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:20.749598 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:20.749912 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:20.749917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:20.749922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:20.750160 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:20.750257 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:20.750808 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 14:26:20.750892 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:26:20.751812 8911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:20.740152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:52949)
0620 14:26:20.740527 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:20.740563 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:20.741794 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:20.742305 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:20.742311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:20.742607 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:20.743000 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:20.743012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.743761 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.743771 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:20.744571 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:20.744580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.745055 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.745058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:20.745251 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:20.745698 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:20.745713 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:20.747098 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:20.749089 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:20.749093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:20.749102 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:20.749403 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:20.749692 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:20.749694 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:20.749695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:20.749817 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:20.750268 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:20.750275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:20.750555 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 14:26:20.750765 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 14:26:20.751064 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260620 14:26:20.752808 8909 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41215
I20260620 14:26:20.753016 8909 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:20.753448 8909 heartbeater.cc:511] Master 127.0.98.62:41215 requested a full tablet report, sending...
I20260620 14:26:20.754343 8584 ts_manager.cc:194] Registered new tserver with Master: a089986493ab44be9a877b64fc031eef (127.0.98.2:40681)
I20260620 14:26:20.754853 8584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52949
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:26:20.770339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41215
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:20.870932 8914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:20.871129 8914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:20.871166 8914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:20.874867 8914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:20.874910 8914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:20.874958 8914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:20.879635 8914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:20.880425 8914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:20.881166 8914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:20.889451 8922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:20.889442 8920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:20.889443 8919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:20.889670 8914 server_base.cc:1034] running on GCE node
I20260620 14:26:20.890058 8914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:20.890527 8914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:20.891670 8914 hybrid_clock.cc:648] HybridClock initialized: now 1781965580891654 us; error 29 us; skew 500 ppm
Jun 20 14:26:20 dist-test-slave-pk3j krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:20.894230 8914 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:26:20.894951 8914 webserver.cc:460] Webserver started at http://127.0.98.3:39693/ using document root <none> and password file <none>
I20260620 14:26:20.895421 8914 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:20.895499 8914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:20.895691 8914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:20.897471 8914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "b4a43baa31b44262be54e4516493a210"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "f3f9916eed53bc96e48ad7977aa53bca"
server_key_iv: "85675e943746ea27e3e267839d6da47f"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.897984 8914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b4a43baa31b44262be54e4516493a210"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "f3f9916eed53bc96e48ad7977aa53bca"
server_key_iv: "85675e943746ea27e3e267839d6da47f"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.901120 8914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:26:20.903339 8929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:20.904327 8914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:26:20.904448 8914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "b4a43baa31b44262be54e4516493a210"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "f3f9916eed53bc96e48ad7977aa53bca"
server_key_iv: "85675e943746ea27e3e267839d6da47f"
server_key_version: "encryptionkey@0"
I20260620 14:26:20.904554 8914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:20.914381 8914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:20.917375 8914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:20.917543 8914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:20.918081 8914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:20.918771 8914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:20.918833 8914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:20.918915 8914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:20.918946 8914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:20.926751 8914 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45953
I20260620 14:26:20.926791 9042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45953 every 8 connection(s)
I20260620 14:26:20.927501 8914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:20.935011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 8914
I20260620 14:26:20.935151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:26:20.935405 392 external_mini_cluster.cc:1428] Setting key d9d3bb44c77996bccea0fdbd508f11e0
Jun 20 14:26:20 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:20.939669 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:20.928973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38953 (local address 127.0.98.62:41215)
0620 14:26:20.929096 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:20.929100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:20.929835 (+ 735us) server_negotiation.cc:407] Connection header received
0620 14:26:20.930995 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:20.930998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:20.931042 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:20.931111 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:20.932149 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.932702 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.933777 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.934046 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.934121 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:20.936571 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:20.936576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:20.936578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:20.936605 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:20.938033 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:20.938449 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:20.938451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:20.938453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:20.938529 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:20.938880 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:20.938882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:20.938883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:20.939041 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:20.939113 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:20.939482 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 14:26:20.939565 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:26:20.940336 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:20.929321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:38953)
0620 14:26:20.929698 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:20.929739 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:20.930833 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:20.931291 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:20.931298 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:20.931646 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:20.932037 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:20.932049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.932802 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.932804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:20.933675 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:20.933683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:20.934157 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:20.934160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:20.934281 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:20.934687 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:20.934701 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:20.936391 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:20.938139 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:20.938143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:20.938152 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:20.938371 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:20.938599 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:20.938600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:20.938601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:20.938777 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:20.939124 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:20.939128 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:20.939306 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 14:26:20.939440 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 14:26:20.939632 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":110,"threads_started":1}
I20260620 14:26:20.941303 9043 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41215
I20260620 14:26:20.941505 9043 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:20.941896 9043 heartbeater.cc:511] Master 127.0.98.62:41215 requested a full tablet report, sending...
I20260620 14:26:20.942786 8584 ts_manager.cc:194] Registered new tserver with Master: b4a43baa31b44262be54e4516493a210 (127.0.98.3:45953)
I20260620 14:26:20.943295 8584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38953
I20260620 14:26:20.948766 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 14:26:21.007611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 8646
I20260620 14:26:21.012969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 8780
I20260620 14:26:21.018170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 8914
I20260620 14:26:21.022945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 8552
I20260620 14:26:21.028729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41215
--webserver_interface=127.0.98.62
--webserver_port=33801
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:21.130465 9052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:21.130703 9052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:21.130770 9052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:21.134498 9052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:21.134569 9052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:21.134603 9052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:21.134672 9052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:21.134722 9052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:21.139597 9052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:21.140475 9052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:21.141352 9052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:21.148867 9060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:21.148869 9057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:21.148902 9058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:21.149056 9052 server_base.cc:1034] running on GCE node
I20260620 14:26:21.149654 9052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:21.150578 9052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:21.151753 9052 hybrid_clock.cc:648] HybridClock initialized: now 1781965581151713 us; error 52 us; skew 500 ppm
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:21.154345 9052 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:26:21.155071 9052 webserver.cc:460] Webserver started at http://127.0.98.62:33801/ using document root <none> and password file <none>
I20260620 14:26:21.155544 9052 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:21.155607 9052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:21.158993 9052 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:26:21.160751 9067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:21.161878 9052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:21.161990 9052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "260cd2b38b144a3196fa4fb283e0a3fc"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "845860e855a165f4446d304ba044bae7"
server_key_iv: "4092237fe8a5c8648ec231045061a21f"
server_key_version: "encryptionkey@0"
I20260620 14:26:21.162443 9052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:21.175035 9052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:21.177521 9052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:21.177690 9052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:21.184023 9052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41215
I20260620 14:26:21.184084 9119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41215 every 8 connection(s)
I20260620 14:26:21.184929 9052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:21.188611 9120 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:26:21.190093 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc: Bootstrap starting.
I20260620 14:26:21.193557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 9052
I20260620 14:26:21.193949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42263
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41215
--webserver_port=42801
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:26:21.205345 9120 log.cc:824] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:21.210964 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:26:21.211292 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc: Bootstrap complete.
I20260620 14:26:21.215188 9120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } }
I20260620 14:26:21.215576 9120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 260cd2b38b144a3196fa4fb283e0a3fc, State: Initialized, Role: FOLLOWER
I20260620 14:26:21.216064 9120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } }
I20260620 14:26:21.216200 9120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:21.216286 9120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:21.216409 9120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:26:21.218258 9120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } }
I20260620 14:26:21.218560 9120 leader_election.cc:302] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 260cd2b38b144a3196fa4fb283e0a3fc; no voters:
I20260620 14:26:21.218849 9120 leader_election.cc:288] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:26:21.218912 9124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:26:21.219105 9124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [term 2 LEADER]: Becoming Leader. State: Replica: 260cd2b38b144a3196fa4fb283e0a3fc, State: Running, Role: LEADER
I20260620 14:26:21.219452 9124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } }
I20260620 14:26:21.219962 9120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:21.220367 9126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 260cd2b38b144a3196fa4fb283e0a3fc. Latest consensus state: current_term: 2 leader_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } } }
I20260620 14:26:21.220487 9126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:21.220685 9125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "260cd2b38b144a3196fa4fb283e0a3fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41215 } } }
I20260620 14:26:21.220810 9125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:21.220957 9128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:21.223891 9128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:21.224465 9128 catalog_manager.cc:1259] Loaded cluster ID: 2b3a9f74a74348fa90553f2535cfa6dc
I20260620 14:26:21.224522 9128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:21.226737 9128 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:21.227368 9128 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 260cd2b38b144a3196fa4fb283e0a3fc: Loaded TSK: 0
I20260620 14:26:21.228505 9128 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:26:21.299965 9122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:21.300206 9122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:21.300275 9122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:21.303994 9122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:21.304071 9122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:21.304168 9122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:21.309058 9122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42801
--tserver_master_addrs=127.0.98.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:21.309907 9122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:21.310781 9122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:21.320300 9147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:21.320480 9146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:21.320629 9149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:21.320791 9122 server_base.cc:1034] running on GCE node
I20260620 14:26:21.321183 9122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:21.321882 9122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:21.323086 9122 hybrid_clock.cc:648] HybridClock initialized: now 1781965581323029 us; error 69 us; skew 500 ppm
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:21.326187 9122 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 14:26:21.326998 9122 webserver.cc:460] Webserver started at http://127.0.98.1:42801/ using document root <none> and password file <none>
I20260620 14:26:21.327505 9122 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:21.327615 9122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:21.331621 9122 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 14:26:21.333714 9156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:21.334859 9122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:21.334966 9122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "6c7491a183e3462d87dc306e14e6183b"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "18799280a1a99e3af70a4b5a257e2494"
server_key_iv: "24fadb8d8f2d515062ecbbbddc0efc78"
server_key_version: "encryptionkey@0"
I20260620 14:26:21.335456 9122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:21.352056 9122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:21.354700 9122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:21.354866 9122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:21.355377 9122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:21.356061 9122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:21.356105 9122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:21.356159 9122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:21.356187 9122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:21.364313 9122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42263
I20260620 14:26:21.364348 9269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42263 every 8 connection(s)
I20260620 14:26:21.365288 9122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:26:21.369432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 9122
I20260620 14:26:21.369834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40681
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41215
--webserver_port=46239
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:26:21.377368 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:21.367156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45607 (local address 127.0.98.62:41215)
0620 14:26:21.367620 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:21.367640 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:21.367862 (+ 222us) server_negotiation.cc:407] Connection header received
0620 14:26:21.369262 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:21.369280 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:21.369584 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:21.369907 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:21.370962 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.371908 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.372977 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.373436 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.373593 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.376674 (+ 3081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45607: BlockingRecv error: recv got EOF from 127.0.98.1:45607 (error 108)
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":109,"threads_started":1}
I20260620 14:26:21.377378 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:21.367251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:45607)
0620 14:26:21.367717 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:21.367754 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:21.368880 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:21.369979 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:21.369987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:21.370358 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:21.370746 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.370758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.372061 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.372064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.372855 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.372864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.373578 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.373582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.373745 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.374254 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:21.374268 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:21.376505 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":139,"threads_started":1}
W20260620 14:26:21.377765 9270 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41215 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41215: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:21.382591 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:21.378206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:34463)
0620 14:26:21.378301 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:21.378316 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:21.378388 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:21.378611 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:21.378614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:21.378748 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:21.378820 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.378824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.379450 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.379454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.379928 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.379935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.380327 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.380331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.380412 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.380800 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:21.380811 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:21.382470 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:26:21.382700 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:21.378262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34463 (local address 127.0.98.62:41215)
0620 14:26:21.378373 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:21.378376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:21.378389 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:21.378481 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:21.378484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:21.378523 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:21.378601 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:21.378922 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.379353 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.380034 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.380223 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.380336 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.382573 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34463: BlockingRecv error: recv got EOF from 127.0.98.1:34463 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:21.387717 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:21.383234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:45449)
0620 14:26:21.383452 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:21.383464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:21.383543 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:21.383741 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:21.383743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:21.383884 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:21.383940 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.383944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.384585 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.384588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.385059 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.385065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.385530 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.385533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.385608 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.386027 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:21.386037 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:21.387596 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156}
W20260620 14:26:21.387794 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:21.383307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45449 (local address 127.0.98.62:41215)
0620 14:26:21.383426 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:21.383430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:21.383520 (+ 90us) server_negotiation.cc:407] Connection header received
0620 14:26:21.383611 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:21.383615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:21.383654 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:21.383728 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:21.384040 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.384480 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.385167 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.385430 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.385529 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.387695 (+ 2166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45449: BlockingRecv error: recv got EOF from 127.0.98.1:45449 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 14:26:21.387888 9270 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:26:21.472666 9274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:21.472891 9274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:21.472940 9274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:21.476600 9274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:21.476644 9274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:21.476737 9274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:21.481509 9274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46239
--tserver_master_addrs=127.0.98.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:21.482295 9274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:21.483132 9274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:21.491539 9282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:21.491534 9279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:21.491534 9280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:21.491791 9274 server_base.cc:1034] running on GCE node
I20260620 14:26:21.492126 9274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:21.492703 9274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:21.493882 9274 hybrid_clock.cc:648] HybridClock initialized: now 1781965581493862 us; error 38 us; skew 500 ppm
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:21.496166 9274 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 14:26:21.496860 9274 webserver.cc:460] Webserver started at http://127.0.98.2:46239/ using document root <none> and password file <none>
I20260620 14:26:21.497424 9274 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:21.497521 9274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:21.500672 9274 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:26:21.502489 9289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:21.503549 9274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:26:21.503679 9274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "a089986493ab44be9a877b64fc031eef"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "c6cf1b3e92b2fc8157a233a91a66fbb7"
server_key_iv: "3a12bdb0fb8158ce956d89fcf3b6f6c2"
server_key_version: "encryptionkey@0"
I20260620 14:26:21.504127 9274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:21.516804 9274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:21.519271 9274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:21.519454 9274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:21.520010 9274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:21.520793 9274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:21.520864 9274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:21.520915 9274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:21.520964 9274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:21.529037 9274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40681
I20260620 14:26:21.529076 9402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40681 every 8 connection(s)
I20260620 14:26:21.529925 9274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:21.535173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 9274
I20260620 14:26:21.535637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45953
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41215
--webserver_port=39693
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:26:21.540524 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:21.531452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49671 (local address 127.0.98.62:41215)
0620 14:26:21.531585 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:21.531589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:21.532332 (+ 743us) server_negotiation.cc:407] Connection header received
0620 14:26:21.533488 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:21.533490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:21.533535 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:21.533601 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:21.534530 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.535005 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.536630 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.536856 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.536987 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.540358 (+ 3371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49671: BlockingRecv error: recv got EOF from 127.0.98.2:49671 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:26:21.540855 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:21.531810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:49671)
0620 14:26:21.532193 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:21.532229 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:21.533316 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:21.533734 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:21.533739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:21.534014 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:21.534403 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.534424 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.535131 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.535135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.536499 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.536513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.537042 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.537047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.537192 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.537774 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:21.537790 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:21.540181 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":112,"threads_started":1}
W20260620 14:26:21.541443 9403 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41215 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41215: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:21.546518 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:21.541849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:54385)
0620 14:26:21.541939 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:21.541951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:21.542019 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:21.542221 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:21.542224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:21.542372 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:21.542449 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.542454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.543111 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.543117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.543658 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.543665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.544091 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.544094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.544182 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.544631 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:21.544650 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:21.546393 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 14:26:21.546595 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:21.541929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54385 (local address 127.0.98.62:41215)
0620 14:26:21.542023 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:21.542027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:21.542040 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:21.542084 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:21.542088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:21.542128 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:21.542208 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:21.542552 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.543011 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.543770 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.543983 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.544089 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.546490 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54385: BlockingRecv error: recv got EOF from 127.0.98.2:54385 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:21.551934 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:21.547206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:46971)
0620 14:26:21.547310 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:21.547324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:21.547390 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:21.547594 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:21.547596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:21.547705 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:21.547763 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.547767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.548697 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.548699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.549164 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.549170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.549749 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.549752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.549804 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.550209 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:21.550220 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:21.551862 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:26:21.552098 9403 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:26:21.552296 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:21.547274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46971 (local address 127.0.98.62:41215)
0620 14:26:21.547372 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:21.547375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:21.547389 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:26:21.547466 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:21.547469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:21.547504 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:21.547575 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:21.547857 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.548599 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.549312 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.549628 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.549740 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.552026 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46971: BlockingRecv error: recv got EOF from 127.0.98.2:46971 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 14:26:21.638401 9406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:21.638600 9406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:21.638639 9406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:21.642336 9406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:21.642383 9406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:21.642432 9406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:21.647015 9406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39693
--tserver_master_addrs=127.0.98.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:21.647756 9406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:21.648571 9406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:21.656913 9412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:21.657047 9411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:21.656944 9414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:21.656971 9406 server_base.cc:1034] running on GCE node
I20260620 14:26:21.657478 9406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:21.657956 9406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:21.659104 9406 hybrid_clock.cc:648] HybridClock initialized: now 1781965581659036 us; error 86 us; skew 500 ppm
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965581, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:21.661592 9406 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 14:26:21.662320 9406 webserver.cc:460] Webserver started at http://127.0.98.3:39693/ using document root <none> and password file <none>
I20260620 14:26:21.662774 9406 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:21.662835 9406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:21.666050 9406 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:21.667869 9421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:21.668933 9406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:21.669049 9406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "b4a43baa31b44262be54e4516493a210"
format_stamp: "Formatted at 2026-06-20 14:26:20 on dist-test-slave-pk3j"
server_key: "f3f9916eed53bc96e48ad7977aa53bca"
server_key_iv: "85675e943746ea27e3e267839d6da47f"
server_key_version: "encryptionkey@0"
I20260620 14:26:21.669548 9406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:21.692163 9406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:21.694623 9406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:21.694794 9406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:21.695294 9406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:21.695976 9406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:21.696019 9406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:21.696081 9406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:21.696117 9406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:21.704282 9406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45953
I20260620 14:26:21.704306 9534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45953 every 8 connection(s)
I20260620 14:26:21.705120 9406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:21.711858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 9406
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:26:21.715451 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:21.706737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46105 (local address 127.0.98.62:41215)
0620 14:26:21.706854 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:21.706857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:21.707597 (+ 740us) server_negotiation.cc:407] Connection header received
0620 14:26:21.708900 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:21.708903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:21.708958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:21.709060 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:21.710138 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.710638 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.711750 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.712007 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.712123 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.715322 (+ 3199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46105: BlockingRecv error: recv got EOF from 127.0.98.3:46105 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:26:21.715881 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:21.707053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:46105)
0620 14:26:21.707447 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:21.707515 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:21.708693 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:21.709277 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:21.709285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:21.709594 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:21.710004 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.710015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.710760 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.710764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.711620 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.711628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.712151 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.712155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.712325 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.712899 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:21.712928 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:21.715170 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
W20260620 14:26:21.716430 9535 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41215 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41215: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:21.721565 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:21.712800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.0.98.62:41215)
0620 14:26:21.713038 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:21.713042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:21.713053 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:26:21.713087 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:21.713089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:21.713119 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:21.713258 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:21.713946 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.714581 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.715452 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.715743 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.715836 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.717424 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:21.717434 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:21.717447 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:21.717472 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:21.719316 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:21.719758 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:21.719762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:21.719766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:21.719928 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:21.720269 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:21.720271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:21.720273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:21.720653 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:21.720842 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:21.721099 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 14:26:21.721464 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":64,"threads_started":1}
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:21.722295 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:21.716850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:57999)
0620 14:26:21.716945 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:21.716976 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:21.717048 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:21.717288 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:21.717291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:21.717434 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:21.717509 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.717514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.718428 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.718434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.719286 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.719296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.719853 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.719856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.719958 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.720455 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:21.720466 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:21.722190 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:26:21.722386 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:21.716898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57999 (local address 127.0.98.62:41215)
0620 14:26:21.717017 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:21.717021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:21.717034 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:21.717153 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:21.717156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:21.717190 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:21.717278 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:21.717616 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.718322 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.719424 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.719750 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.719835 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.722297 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57999: BlockingRecv error: recv got EOF from 127.0.98.3:57999 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:21 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:21.727083 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:21.722788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:40947)
0620 14:26:21.722866 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:21.722881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:21.722949 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:21.723149 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:21.723151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:21.723296 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:21.723350 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.723355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.724122 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.724125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.724569 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:21.724575 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.724971 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.724974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:21.725042 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.725467 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:21.725479 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:21.726949 (+ 1470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 14:26:21.727254 9272 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:21.722841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40947 (local address 127.0.98.62:41215)
0620 14:26:21.722957 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:21.722961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:21.722974 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:21.723021 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:21.723023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:21.723053 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:21.723137 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:21.723452 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.723946 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.724675 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:21.724875 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:21.724970 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:21.727059 (+ 2089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40947: BlockingRecv error: recv got EOF from 127.0.98.3:40947 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 14:26:21.727416 9535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:26:22 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:22 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:22.393945 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:22.388474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:47721)
0620 14:26:22.388740 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:22.388759 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:22.388854 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:22.389074 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:22.389076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:22.389297 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:22.389457 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:22.389463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.390204 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.390211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:22.391109 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:22.391119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.391575 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.391578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:22.391649 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:22.392082 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:22.392093 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:22.393839 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":59,"threads_started":1}
W20260620 14:26:22.394133 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:22.388640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47721 (local address 127.0.98.62:41215)
0620 14:26:22.388886 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:22.388890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:22.388901 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:26:22.388931 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:22.388933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:22.388969 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:22.389040 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:22.389574 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.390117 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.391230 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.391481 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.391562 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:22.393969 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47721: BlockingRecv error: recv got EOF from 127.0.98.1:47721 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
Jun 20 14:26:22 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:22 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:22.557615 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:22.552658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:51031)
0620 14:26:22.552916 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:22.552932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:22.553023 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:22.553281 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:22.553284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:22.553442 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:22.553548 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:22.553554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.554215 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.554217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:22.554751 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:22.554757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.555132 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.555135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:22.555197 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:22.555714 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:22.555726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:22.557497 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
W20260620 14:26:22.557775 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:22.552805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51031 (local address 127.0.98.62:41215)
0620 14:26:22.552967 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:22.552972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:22.552986 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:26:22.553106 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:22.553108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:22.553147 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:22.553267 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:22.553645 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.554129 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.554849 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.555045 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.555113 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:22.557626 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51031: BlockingRecv error: recv got EOF from 127.0.98.2:51031 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 14:26:22 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:22 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:22.733386 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:22.727930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:59221)
0620 14:26:22.728205 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:22.728217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:22.728331 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:22.728589 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:22.728592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:22.728784 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:22.728886 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:22.728891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.729701 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.729705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:22.730244 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:22.730252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.730690 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.730693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:22.730786 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:22.731316 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:22.731327 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:22.733183 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
W20260620 14:26:22.733448 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:22.728006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59221 (local address 127.0.98.62:41215)
0620 14:26:22.728145 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:22.728149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:22.728285 (+ 136us) server_negotiation.cc:407] Connection header received
0620 14:26:22.728431 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:22.728434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:22.728487 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:22.728571 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:22.729006 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.729590 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.730360 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:22.730580 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:22.730695 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:22.733335 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59221: BlockingRecv error: recv got EOF from 127.0.98.3:59221 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 14:26:23 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:23 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:23.399945 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:23.394632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:49513)
0620 14:26:23.394878 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:23.394891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:23.394975 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:23.395236 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:23.395238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:23.395388 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:23.395475 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:23.395480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.396119 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.396121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:23.396613 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:23.396619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.397376 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.397378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:23.397444 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:23.397919 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:23.397930 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:23.399767 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
W20260620 14:26:23.400003 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:23.394702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49513 (local address 127.0.98.62:41215)
0620 14:26:23.394901 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:23.394905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:23.394936 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:26:23.395083 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:23.395085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:23.395128 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:23.395211 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:23.395565 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.396035 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.396937 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.397257 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.397347 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:23.399885 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49513: BlockingRecv error: recv got EOF from 127.0.98.1:49513 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
Jun 20 14:26:23 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:23 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:23.563669 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:23.558345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:57499)
0620 14:26:23.558620 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:23.558631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:23.558747 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:23.559009 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:23.559012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:23.559198 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:23.559286 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:23.559292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.560022 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.560026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:23.560565 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:23.560573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.561009 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.561012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:23.561106 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:23.561601 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:23.561612 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:23.563508 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
W20260620 14:26:23.563720 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:23.558425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57499 (local address 127.0.98.62:41215)
0620 14:26:23.558547 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:23.558552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:23.558699 (+ 147us) server_negotiation.cc:407] Connection header received
0620 14:26:23.558844 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:23.558847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:23.558893 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:23.558967 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:23.559410 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.559915 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.560680 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.560901 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.561004 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:23.563621 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57499: BlockingRecv error: recv got EOF from 127.0.98.2:57499 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:26:23 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:23 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:23.739691 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:23.734157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:59929)
0620 14:26:23.734428 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:23.734444 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:23.734546 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:23.734822 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:23.734825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:23.735021 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:23.735105 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:23.735110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.735847 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.735853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:23.736414 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:23.736421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.736866 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.736869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:23.736951 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:23.737527 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:23.737541 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:23.739503 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
W20260620 14:26:23.739715 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:23.734298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59929 (local address 127.0.98.62:41215)
0620 14:26:23.734428 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:23.734434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:23.734501 (+ 67us) server_negotiation.cc:407] Connection header received
0620 14:26:23.734647 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:23.734650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:23.734698 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:23.734787 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:23.735216 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.735731 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.736531 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:23.736755 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:23.736865 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:23.739622 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59929: BlockingRecv error: recv got EOF from 127.0.98.3:59929 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 14:26:24 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:24 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:24.405714 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:24.400713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:48231)
0620 14:26:24.401009 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:24.401022 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:24.401104 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:24.401365 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:24.401367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:24.401531 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:24.401617 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:24.401622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.402254 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.402256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:24.402799 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:24.402806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.403194 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.403196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:24.403255 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:24.403698 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:24.403708 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:24.405612 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":65,"threads_started":1}
W20260620 14:26:24.405862 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:24.400784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48231 (local address 127.0.98.62:41215)
0620 14:26:24.401041 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:24.401045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:24.401061 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:26:24.401188 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:24.401190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:24.401254 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:24.401326 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:24.401706 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.402170 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.402901 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.403107 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.403177 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:24.405732 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48231: BlockingRecv error: recv got EOF from 127.0.98.1:48231 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":128,"threads_started":1}
Jun 20 14:26:24 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:24 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:24.569571 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:24.564430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:45891)
0620 14:26:24.564675 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:24.564693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:24.564803 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:24.565063 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:24.565065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:24.565271 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:24.565359 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:24.565364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.566087 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.566089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:24.566686 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:24.566694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.567093 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.567095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:24.567180 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:24.567631 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:24.567642 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:24.569457 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
W20260620 14:26:24.569697 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:24.564486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45891 (local address 127.0.98.62:41215)
0620 14:26:24.564636 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:24.564640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:24.564759 (+ 119us) server_negotiation.cc:407] Connection header received
0620 14:26:24.564919 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:24.564922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:24.564969 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:24.565045 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:24.565491 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.566003 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.566784 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.566998 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.567075 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:24.569581 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45891: BlockingRecv error: recv got EOF from 127.0.98.2:45891 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 14:26:24 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:24 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:24.745615 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:24.740294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:42597)
0620 14:26:24.740533 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:24.740545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:24.740632 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:24.740845 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:24.740847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:24.741005 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:24.741094 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:24.741099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.741874 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.741877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:24.742748 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:24.742759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.743204 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.743207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:24.743262 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:24.743702 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:24.743713 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:24.745499 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":62,"threads_started":1}
W20260620 14:26:24.745723 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:24.740375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42597 (local address 127.0.98.62:41215)
0620 14:26:24.740542 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:24.740545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:24.740611 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:26:24.740707 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:24.740709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:24.740749 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:24.740811 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:24.741187 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.741792 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.742866 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:24.743105 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:24.743179 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:24.745614 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42597: BlockingRecv error: recv got EOF from 127.0.98.3:42597 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 14:26:25 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:25 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:25.415133 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:25.406442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:38549)
0620 14:26:25.406702 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:25.406727 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:25.406814 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:25.407078 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:25.407080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:25.407258 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:25.407351 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:25.407356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.408142 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.408144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:25.408725 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:25.408732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.409169 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.409173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:25.409407 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:25.413174 (+ 3767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:25.413187 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:25.415004 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":105,"threads_started":1}
W20260620 14:26:25.415331 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:25.406490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38549 (local address 127.0.98.62:41215)
0620 14:26:25.406717 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:25.406721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:25.406791 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:26:25.406930 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:25.406932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:25.406973 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:25.407048 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:25.407444 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.408031 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.408823 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.409055 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.409166 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:25.415164 (+ 5998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38549: BlockingRecv error: recv got EOF from 127.0.98.1:38549 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
Jun 20 14:26:25 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:25 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:25.575477 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:25.570250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:52849)
0620 14:26:25.570534 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:25.570546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:25.570636 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:25.570855 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:25.570857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:25.571017 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:25.571118 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:25.571123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.571850 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.571853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:25.572430 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:25.572438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.572935 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.572938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:25.573002 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:25.573579 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:25.573592 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:25.575366 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
W20260620 14:26:25.575595 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:25.570292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52849 (local address 127.0.98.62:41215)
0620 14:26:25.570455 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:25.570460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:25.570616 (+ 156us) server_negotiation.cc:407] Connection header received
0620 14:26:25.570713 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:25.570715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:25.570758 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:25.570821 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:25.571255 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.571756 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.572562 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.572825 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.572910 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:25.575486 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52849: BlockingRecv error: recv got EOF from 127.0.98.2:52849 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:26:25 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:25 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:25.751703 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:25.746474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:40695)
0620 14:26:25.746711 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:25.746723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:25.746810 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:25.747023 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:25.747025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:25.747185 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:25.747297 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:25.747301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.748006 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.748011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:25.748770 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:25.748780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.749316 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.749318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:25.749377 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:25.749821 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:25.749831 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:25.751584 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":62,"threads_started":1}
W20260620 14:26:25.751842 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:25.746574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40695 (local address 127.0.98.62:41215)
0620 14:26:25.746723 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:25.746726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:25.746791 (+ 65us) server_negotiation.cc:407] Connection header received
0620 14:26:25.746885 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:25.746887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:25.746927 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:25.746991 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:25.747388 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.747888 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.748903 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:25.749146 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:25.749295 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:25.751712 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40695: BlockingRecv error: recv got EOF from 127.0.98.3:40695 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 14:26:26 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:26 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:26.421049 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:26.415878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:47777)
0620 14:26:26.416144 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:26.416157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:26.416245 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:26.416495 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:26.416497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:26.416661 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:26.416750 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:26.416754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.417595 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.417597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:26.418159 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:26.418166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.418547 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.418549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:26.418609 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:26.419058 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:26.419068 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:26.420920 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":65,"threads_started":1}
W20260620 14:26:26.421164 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:26.415952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47777 (local address 127.0.98.62:41215)
0620 14:26:26.416210 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:26.416214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:26.416226 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:26:26.416346 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:26.416348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:26.416388 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:26.416461 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:26.416860 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.417499 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.418248 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.418459 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.418534 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:26.420998 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47777: BlockingRecv error: recv got EOF from 127.0.98.1:47777 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":71,"threads_started":1}
Jun 20 14:26:26 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:26 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:26.581079 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:26.576250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:37579)
0620 14:26:26.576539 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:26.576552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:26.576632 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:26.576850 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:26.576852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:26.577010 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:26.577102 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:26.577107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.577814 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.577817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:26.578368 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:26.578375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.578802 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.578804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:26.578863 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:26.579313 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:26.579324 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:26.580962 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
W20260620 14:26:26.581200 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:26.576418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37579 (local address 127.0.98.62:41215)
0620 14:26:26.576566 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:26.576570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:26.576596 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:26:26.576710 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:26.576712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:26.576751 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:26.576821 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:26.577193 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.577728 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.578467 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.578710 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.578785 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:26.581088 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37579: BlockingRecv error: recv got EOF from 127.0.98.2:37579 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 14:26:26 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:26 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:26.757717 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:26.752405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:46709)
0620 14:26:26.752645 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:26.752658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:26.752766 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:26.753056 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:26.753060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:26.753323 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:26.753444 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:26.753452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.754350 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.754353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:26.754929 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:26.754937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.755310 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.755312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:26.755369 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:26.755815 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:26.755825 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:26.757604 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
W20260620 14:26:26.757951 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:26.752484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46709 (local address 127.0.98.62:41215)
0620 14:26:26.752625 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:26.752630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:26.752731 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:26:26.752871 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:26.752875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:26.752928 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:26.753022 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:26.753571 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.754250 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.755025 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:26.755224 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:26.755293 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:26.757734 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46709: BlockingRecv error: recv got EOF from 127.0.98.3:46709 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 14:26:27 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:27 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:27.427150 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:27.421921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:60247)
0620 14:26:27.422239 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:27.422256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:27.422393 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:27.422667 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:27.422670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:27.422866 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:27.422970 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:27.422977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.423684 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.423688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:27.424204 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:27.424212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.424643 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.424646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:27.424729 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:27.425294 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:27.425310 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:27.426991 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":75,"threads_started":1}
W20260620 14:26:27.427233 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:27.422018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60247 (local address 127.0.98.62:41215)
0620 14:26:27.422239 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:27.422245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:27.422338 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:26:27.422497 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:27.422500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:27.422551 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:27.422641 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:27.423097 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.423584 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.424323 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.424538 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.424639 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:27.427112 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60247: BlockingRecv error: recv got EOF from 127.0.98.1:60247 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
Jun 20 14:26:27 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:27 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:27.587061 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:27.582086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:58141)
0620 14:26:27.582338 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:27.582350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:27.582437 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:27.582609 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:27.582611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:27.582774 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:27.582862 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:27.582867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.583497 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.583499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:27.583994 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:27.584000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.584415 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.584417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:27.584476 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:27.585002 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:27.585014 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:27.586955 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
W20260620 14:26:27.587298 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:27.582133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58141 (local address 127.0.98.62:41215)
0620 14:26:27.582271 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:27.582275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:27.582409 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:26:27.582496 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:27.582498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:27.582537 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:27.582598 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:27.582946 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.583414 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.584081 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.584316 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.584394 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:27.587079 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58141: BlockingRecv error: recv got EOF from 127.0.98.2:58141 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 14:26:27 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:27 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:27.763064 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:27.758338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:56685)
0620 14:26:27.758606 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:27.758619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:27.758695 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:27.758896 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:27.758898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:27.759055 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:27.759143 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:27.759148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.759829 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.759831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:27.760370 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:27.760377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.760771 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.760774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:27.760829 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:27.761346 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:27.761356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:27.762961 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
W20260620 14:26:27.763216 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:27.758512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56685 (local address 127.0.98.62:41215)
0620 14:26:27.758650 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:27.758654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:27.758665 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:26:27.758759 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:27.758761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:27.758801 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:27.758863 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:27.759236 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.759742 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.760464 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:27.760684 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:27.760754 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:27.763099 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56685: BlockingRecv error: recv got EOF from 127.0.98.3:56685 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 14:26:28 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:28 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:28.432838 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:28.427899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:46501)
0620 14:26:28.428206 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:28.428220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:28.428305 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:28.428551 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:28.428553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:28.428731 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:28.428822 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:28.428827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.429510 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.429513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:28.430023 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:28.430030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.430386 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.430388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:28.430444 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:28.430955 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:28.430966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:28.432736 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":65,"threads_started":1}
W20260620 14:26:28.433030 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:28.428025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46501 (local address 127.0.98.62:41215)
0620 14:26:28.428271 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:28.428274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:28.428287 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:28.428390 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:28.428392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:28.428433 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:28.428507 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:28.428915 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.429425 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.430111 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.430302 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.430367 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:28.432864 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46501: BlockingRecv error: recv got EOF from 127.0.98.1:46501 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
Jun 20 14:26:28 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:28 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:28.593253 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:28.587953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:47707)
0620 14:26:28.588277 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:28.588292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:28.588428 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:28.588715 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:28.588718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:28.588915 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:28.589002 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:28.589008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.589777 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.589782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:28.590306 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:28.590314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.590738 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.590741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:28.590814 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:28.591292 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:28.591304 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:28.593070 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
W20260620 14:26:28.593313 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:28.587992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47707 (local address 127.0.98.62:41215)
0620 14:26:28.588140 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:28.588146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:28.588377 (+ 231us) server_negotiation.cc:407] Connection header received
0620 14:26:28.588544 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:28.588547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:28.588595 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:28.588694 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:28.589116 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.589670 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.590417 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.590637 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.590735 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:28.593187 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47707: BlockingRecv error: recv got EOF from 127.0.98.2:47707 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 14:26:28 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:28 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:28.768839 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:28.763776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:49419)
0620 14:26:28.764060 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:28.764072 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:28.764161 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:28.764365 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:28.764367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:28.764529 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:28.764625 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:28.764630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.765404 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.765407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:28.765980 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:28.765988 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.766403 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.766406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:28.766463 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:28.766919 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:28.766930 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:28.768680 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":103,"threads_started":1}
W20260620 14:26:28.768878 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:28.763860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49419 (local address 127.0.98.62:41215)
0620 14:26:28.763981 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:28.763984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:28.764136 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:26:28.764223 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:28.764225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:28.764267 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:28.764333 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:28.764748 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.765303 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.766072 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:28.766313 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:28.766390 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:28.768786 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49419: BlockingRecv error: recv got EOF from 127.0.98.3:49419 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 14:26:29 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:29 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:29.438779 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:29.433650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:39359)
0620 14:26:29.433903 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:29.433915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:29.433993 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:29.434210 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:29.434212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:29.434368 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:29.434455 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:29.434459 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.435115 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.435117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:29.435668 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:29.435677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.436218 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.436223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:29.436312 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:29.436804 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:29.436815 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:29.438660 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
W20260620 14:26:29.438921 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:29.433752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39359 (local address 127.0.98.62:41215)
0620 14:26:29.433989 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:29.433992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:29.434003 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:26:29.434071 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:29.434073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:29.434110 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:29.434180 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:29.434563 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.435031 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.435782 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.436105 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.436203 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:29.438799 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39359: BlockingRecv error: recv got EOF from 127.0.98.1:39359 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":109,"threads_started":1}
Jun 20 14:26:29 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:29 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:29.599386 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:29.594064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:44007)
0620 14:26:29.594352 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:29.594370 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:29.594489 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:29.594747 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:29.594750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:29.594943 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:29.595028 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:29.595033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.595896 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.595901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:29.596449 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:29.596456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.596913 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.596916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:29.596997 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:29.597518 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:29.597530 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:29.599235 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
W20260620 14:26:29.599442 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:29.594116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44007 (local address 127.0.98.62:41215)
0620 14:26:29.594280 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:29.594285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:29.594440 (+ 155us) server_negotiation.cc:407] Connection header received
0620 14:26:29.594587 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:29.594590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:29.594637 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:29.594730 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:29.595140 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.595788 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.596563 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.596805 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.596911 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:29.599335 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44007: BlockingRecv error: recv got EOF from 127.0.98.2:44007 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 14:26:29 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:29 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:29.775041 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:29.769669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:43991)
0620 14:26:29.769931 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:29.769943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:29.770056 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:29.770286 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:29.770288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:29.770508 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:29.770623 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:29.770630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.771595 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.771598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:29.772191 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:29.772201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.772571 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.772573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:29.772630 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:29.773156 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:29.773171 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:29.774937 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
W20260620 14:26:29.775153 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:29.769733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43991 (local address 127.0.98.62:41215)
0620 14:26:29.769849 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:29.769852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:29.770003 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:26:29.770128 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:29.770130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:29.770172 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:29.770262 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:29.770726 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.771490 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.772286 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:29.772483 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:29.772556 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:29.775052 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43991: BlockingRecv error: recv got EOF from 127.0.98.3:43991 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 14:26:30 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:30 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:30.448951 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:30.439504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:49947)
0620 14:26:30.439816 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:30.439829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:30.439914 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:30.440145 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:30.440148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:30.440330 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:30.440430 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:30.440438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.441127 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.441129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:30.441684 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:30.441692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.442068 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.442071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:30.442127 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:30.446688 (+ 4561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:30.446700 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:30.448834 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
W20260620 14:26:30.449077 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:30.439611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49947 (local address 127.0.98.62:41215)
0620 14:26:30.439865 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:30.439869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:30.439881 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:26:30.439993 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:30.439996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:30.440035 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:30.440109 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:30.440532 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.441033 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.441775 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.441977 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.442051 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:30.448955 (+ 6904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49947: BlockingRecv error: recv got EOF from 127.0.98.1:49947 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
Jun 20 14:26:30 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:30 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:30.604969 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:30.600216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:40011)
0620 14:26:30.600478 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:30.600491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:30.600577 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:30.600782 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:30.600784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:30.600942 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:30.601047 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:30.601052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.601720 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.601722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:30.602264 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:30.602271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.602646 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.602648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:30.602715 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:30.603183 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:30.603194 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:30.604824 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
W20260620 14:26:30.605034 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:30.600284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40011 (local address 127.0.98.62:41215)
0620 14:26:30.600417 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:30.600420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:30.600552 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:26:30.600639 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:30.600641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:30.600680 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:30.600750 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:30.601140 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.601637 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.602361 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.602557 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.602627 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:30.604953 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40011: BlockingRecv error: recv got EOF from 127.0.98.2:40011 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:26:30 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:30 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:30.781373 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:30.775929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:36525)
0620 14:26:30.776217 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:30.776230 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:30.776355 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:30.776624 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:30.776627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:30.776829 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:30.776914 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:30.776919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.777754 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.777759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:30.778328 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:30.778337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.778828 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.778831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:30.778915 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:30.779401 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:30.779412 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:30.781185 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":113,"threads_started":1}
W20260620 14:26:30.781438 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:30.775994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36525 (local address 127.0.98.62:41215)
0620 14:26:30.776132 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:30.776138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:30.776309 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:26:30.776458 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:30.776462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:30.776517 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:30.776605 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:30.777032 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.777644 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.778452 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:30.778700 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:30.778810 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:30.781330 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36525: BlockingRecv error: recv got EOF from 127.0.98.3:36525 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:26:31 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:31 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:31.455086 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:31.449864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:38139)
0620 14:26:31.450125 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:31.450138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:31.450218 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:31.450459 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:31.450461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:31.450642 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:31.450733 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:31.450738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.451555 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.451557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:31.452151 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:31.452160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.452568 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.452571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:31.452633 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:31.453153 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:31.453164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:31.454914 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
W20260620 14:26:31.455168 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:31.449922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38139 (local address 127.0.98.62:41215)
0620 14:26:31.450157 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:31.450161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:31.450178 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:26:31.450302 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:31.450304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:31.450347 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:31.450424 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:31.450877 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.451446 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.452253 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.452460 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.452558 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:31.455046 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38139: BlockingRecv error: recv got EOF from 127.0.98.1:38139 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":96,"threads_started":1}
Jun 20 14:26:31 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:31 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:31.610924 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:31.605776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:34413)
0620 14:26:31.606025 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:31.606039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:31.606117 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:31.606336 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:31.606338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:31.606504 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:31.606592 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:31.606597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.607372 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.607376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:31.607948 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:31.607956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.608441 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.608444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:31.608539 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:31.608991 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:31.609002 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:31.610741 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
W20260620 14:26:31.610952 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:31.605949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34413 (local address 127.0.98.62:41215)
0620 14:26:31.606104 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:31.606107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:31.606120 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:31.606204 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:31.606206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:31.606244 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:31.606311 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:31.606730 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.607260 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.608070 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.608312 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.608422 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:31.610856 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34413: BlockingRecv error: recv got EOF from 127.0.98.2:34413 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 14:26:31 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:31 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:31.787334 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:31.782152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:40677)
0620 14:26:31.782440 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:31.782453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:31.782537 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:31.782771 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:31.782773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:31.782936 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:31.783025 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:31.783030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.783716 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.783718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:31.784246 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:31.784254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.784847 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.784850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:31.784949 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:31.785532 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:31.785544 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:31.787229 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":112,"threads_started":1}
W20260620 14:26:31.787495 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:31.782303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40677 (local address 127.0.98.62:41215)
0620 14:26:31.782443 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:31.782447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:31.782517 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:26:31.782625 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:31.782627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:31.782669 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:31.782734 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:31.783136 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.783629 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.784373 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:31.784714 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:31.784846 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:31.787348 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40677: BlockingRecv error: recv got EOF from 127.0.98.3:40677 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:26:32 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:32 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:32.460994 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:32.455955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:59791)
0620 14:26:32.456240 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:32.456252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:32.456332 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:32.456562 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:32.456564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:32.456725 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:32.456830 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:32.456835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.457551 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.457554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:32.458112 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:32.458119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.458521 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.458524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:32.458583 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:32.459138 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:32.459149 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:32.460886 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
W20260620 14:26:32.461151 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:32.456053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59791 (local address 127.0.98.62:41215)
0620 14:26:32.456288 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:32.456291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:32.456303 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:26:32.456409 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:32.456411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:32.456456 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:32.456528 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:32.456923 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.457454 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.458216 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.458426 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.458503 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:32.461017 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59791: BlockingRecv error: recv got EOF from 127.0.98.1:59791 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
Jun 20 14:26:32 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:32 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:32.616672 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:32.611710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:54537)
0620 14:26:32.611970 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:32.611982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:32.612074 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:32.612290 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:32.612292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:32.612472 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:32.612572 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:32.612577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.613300 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.613302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:32.613860 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:32.613868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.614282 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.614284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:32.614346 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:32.614822 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:32.614833 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:32.616555 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
W20260620 14:26:32.616783 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:32.611806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54537 (local address 127.0.98.62:41215)
0620 14:26:32.611979 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:32.611983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:32.612030 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:26:32.612149 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:32.612151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:32.612197 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:32.612254 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:32.612669 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.613159 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.613958 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.614174 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.614250 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:32.616674 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54537: BlockingRecv error: recv got EOF from 127.0.98.2:54537 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 14:26:32 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:32 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:32.793514 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:32.788144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:50389)
0620 14:26:32.788448 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:32.788461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:32.788556 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:32.788776 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:32.788778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:32.788956 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:32.789045 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:32.789049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.789734 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.789737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:32.790535 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:32.790545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.791104 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.791107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:32.791167 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:32.791640 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:32.791651 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:32.793404 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":108,"threads_started":1}
W20260620 14:26:32.793736 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:32.788187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50389 (local address 127.0.98.62:41215)
0620 14:26:32.788338 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:32.788343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:32.788531 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:26:32.788632 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:32.788634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:32.788677 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:32.788741 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:32.789134 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.789636 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.790663 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:32.791001 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:32.791085 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:32.793514 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50389: BlockingRecv error: recv got EOF from 127.0.98.3:50389 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:26:33 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:33 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:33.466702 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:33.461782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:35037)
0620 14:26:33.462036 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:33.462049 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:33.462138 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:33.462355 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:33.462357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:33.462532 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:33.462624 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:33.462629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.463290 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.463292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:33.463824 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:33.463831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.464211 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.464213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:33.464271 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:33.464801 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:33.464812 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:33.466596 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
W20260620 14:26:33.466888 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:33.461819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35037 (local address 127.0.98.62:41215)
0620 14:26:33.462032 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:33.462036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:33.462114 (+ 78us) server_negotiation.cc:407] Connection header received
0620 14:26:33.462210 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:33.462212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:33.462251 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:33.462325 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:33.462715 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.463203 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.463921 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.464123 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.464194 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:33.466722 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35037: BlockingRecv error: recv got EOF from 127.0.98.1:35037 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
Jun 20 14:26:33 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:33 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:33.622681 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:33.617471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:52775)
0620 14:26:33.617710 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:33.617721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:33.617865 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:33.618122 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:33.618125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:33.618325 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:33.618415 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:33.618421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.619173 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.619177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:33.619712 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:33.619720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.620262 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.620266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:33.620358 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:33.620769 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:33.620779 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:33.622523 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
W20260620 14:26:33.622751 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:33.617503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52775 (local address 127.0.98.62:41215)
0620 14:26:33.617641 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:33.617656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:33.617809 (+ 153us) server_negotiation.cc:407] Connection header received
0620 14:26:33.617962 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:33.617965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:33.618016 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:33.618103 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:33.618527 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.619058 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.619832 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.620144 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.620247 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:33.622638 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52775: BlockingRecv error: recv got EOF from 127.0.98.2:52775 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:26:33 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:33 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:33.799753 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:33.794356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:34303)
0620 14:26:33.794618 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:33.794630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:33.794720 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:33.794924 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:33.794926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:33.795102 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:33.795209 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:33.795215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.796194 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.796198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:33.796810 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:33.796819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.797247 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.797251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:33.797324 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:33.797864 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:33.797877 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:33.799649 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":110,"threads_started":1}
W20260620 14:26:33.799890 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:33.794407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34303 (local address 127.0.98.62:41215)
0620 14:26:33.794537 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:33.794540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:33.794696 (+ 156us) server_negotiation.cc:407] Connection header received
0620 14:26:33.794781 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:33.794783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:33.794825 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:33.794892 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:33.795316 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.796084 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.796911 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:33.797113 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:33.797192 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:33.799770 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34303: BlockingRecv error: recv got EOF from 127.0.98.3:34303 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:26:34 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:34 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:34.473284 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:34.467362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:47665)
0620 14:26:34.467621 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:34.467642 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:34.467725 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:34.467963 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:34.467965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:34.468136 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:34.468229 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:34.468233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.469231 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.469236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:34.469971 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:34.469981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.470599 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.470602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:34.470731 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:34.471304 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:34.471315 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:34.473110 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
W20260620 14:26:34.473397 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:34.467466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47665 (local address 127.0.98.62:41215)
0620 14:26:34.467731 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:34.467734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:34.467747 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:26:34.467782 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:34.467784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:34.467841 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:34.467921 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:34.468326 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.469050 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.470112 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.470480 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.470574 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:34.473257 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47665: BlockingRecv error: recv got EOF from 127.0.98.1:47665 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
Jun 20 14:26:34 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:34 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:26:34.628885 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:34.623492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34035 (local address 127.0.98.62:41215)
0620 14:26:34.623660 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:34.623664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:34.623738 (+ 74us) server_negotiation.cc:407] Connection header received
0620 14:26:34.623839 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:34.623841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:34.623882 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:34.623951 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:34.624333 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.624850 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.625745 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.625959 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.626035 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:34.628789 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34035: BlockingRecv error: recv got EOF from 127.0.98.2:34035 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:26:34.628969 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:34.623381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:34035)
0620 14:26:34.623661 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:34.623674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:34.623766 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:34.623981 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:34.623983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:34.624154 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:34.624243 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:34.624248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.624935 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.624937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:34.625643 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:34.625654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.626058 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.626060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:34.626117 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:34.626588 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:34.626598 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:34.628670 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":96,"threads_started":1}
Jun 20 14:26:34 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:34 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:34.805374 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:34.800510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:33731)
0620 14:26:34.800794 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:34.800816 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:34.800909 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:34.801112 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:34.801114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:34.801300 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:34.801390 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:34.801394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.802084 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.802086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:34.802608 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:34.802617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.803003 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.803006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:34.803061 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:34.803533 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:34.803544 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:34.805264 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
W20260620 14:26:34.805480 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:34.800674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33731 (local address 127.0.98.62:41215)
0620 14:26:34.800786 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:34.800789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:34.800884 (+ 95us) server_negotiation.cc:407] Connection header received
0620 14:26:34.800974 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:34.800976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:34.801017 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:34.801078 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:34.801482 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.801999 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.802700 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:34.802901 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:34.802984 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:34.805358 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33731: BlockingRecv error: recv got EOF from 127.0.98.3:33731 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 14:26:35 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:35 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:35.482973 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:35.473973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:58481)
0620 14:26:35.474236 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:35.474249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:35.474330 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:35.474633 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:35.474635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:35.474792 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:35.474882 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:35.474887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.475576 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.475578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:35.476106 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:35.476121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.476504 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.476506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:35.476565 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:35.480846 (+ 4281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:35.480858 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:35.482805 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
W20260620 14:26:35.483052 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:35.474168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58481 (local address 127.0.98.62:41215)
0620 14:26:35.474434 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:35.474438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:35.474449 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:26:35.474481 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:35.474483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:35.474521 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:35.474600 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:35.474981 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.475487 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.476209 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.476416 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.476486 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:35.482931 (+ 6445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58481: BlockingRecv error: recv got EOF from 127.0.98.1:58481 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
Jun 20 14:26:35 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:35 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:35.635185 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:35.629647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:49807)
0620 14:26:35.629916 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:35.629928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:35.630058 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:35.630310 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:35.630312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:35.630506 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:35.630593 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:35.630599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.631364 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.631369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:35.631924 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:35.631931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.632455 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.632460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:35.632556 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:35.633164 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:35.633179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:35.635016 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
W20260620 14:26:35.635218 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:35.629702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49807 (local address 127.0.98.62:41215)
0620 14:26:35.629858 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:35.629863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:35.630010 (+ 147us) server_negotiation.cc:407] Connection header received
0620 14:26:35.630157 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:35.630159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:35.630211 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:35.630293 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:35.630702 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.631253 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.632066 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.632301 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.632425 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:35.635127 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49807: BlockingRecv error: recv got EOF from 127.0.98.2:49807 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 14:26:35 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:35 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:35.811204 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:35.806084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.3:49611)
0620 14:26:35.806371 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:35.806382 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:35.806471 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:35.806676 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:35.806678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:35.806903 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:35.806993 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:35.806999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.807711 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.807713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:35.808323 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:35.808331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.808732 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.808734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:35.808789 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:35.809273 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:35.809283 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:35.811014 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":78,"threads_started":1}
W20260620 14:26:35.811226 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:35.806133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49611 (local address 127.0.98.62:41215)
0620 14:26:35.806233 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:35.806236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:35.806442 (+ 206us) server_negotiation.cc:407] Connection header received
0620 14:26:35.806536 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:35.806538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:35.806579 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:35.806642 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:35.807072 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.807599 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.808421 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:35.808638 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:35.808717 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:35.811130 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49611: BlockingRecv error: recv got EOF from 127.0.98.3:49611 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 14:26:36 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:36 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:36.489364 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:36.483927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.1:49205)
0620 14:26:36.484294 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:36.484312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:36.484422 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:36.484744 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:36.484748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:36.484953 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:36.485064 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:36.485070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:36.485817 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:36.485819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:36.486355 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:36.486362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:36.486756 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:36.486758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:36.486834 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:36.487403 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:36.487423 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:36.489174 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":85,"threads_started":1}
W20260620 14:26:36.489511 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:36.484042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49205 (local address 127.0.98.62:41215)
0620 14:26:36.484298 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:36.484322 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:36.484391 (+ 69us) server_negotiation.cc:407] Connection header received
0620 14:26:36.484515 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:36.484518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:36.484572 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:36.484692 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:36.485162 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:36.485733 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:36.486458 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:36.486668 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:36.486739 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:36.489361 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49205: BlockingRecv error: recv got EOF from 127.0.98.1:49205 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
Jun 20 14:26:36 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:26:36 dist-test-slave-pk3j krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:26:36.641430 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:36.636103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41215 (local address 127.0.98.2:52373)
0620 14:26:36.636386 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:36.636399 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:36.636522 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:36.636778 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:36.636781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:36.636982 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:36.637070 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:36.637075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:36.637847 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:36.637851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:36.638461 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:36.638469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:36.638976 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:36.638980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:36.639067 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:36.639581 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:36.639593 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:36.641257 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41215: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
W20260620 14:26:36.641462 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:36.636163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52373 (local address 127.0.98.62:41215)
0620 14:26:36.636332 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:36.636336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:36.636473 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:26:36.636618 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:36.636622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:36.636669 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:36.636760 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:36.637185 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:36.637756 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:36.638581 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:36.638846 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:36.638974 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:36.641366 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52373: BlockingRecv error: recv got EOF from 127.0.98.2:52373 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 14:26:36.720816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 9122
I20260620 14:26:36.726183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 9274
I20260620 14:26:36.731196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 9406
I20260620 14:26:36.736150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 9052
2026-06-20T14:26:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17706 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:36Z Disabled control of system clock
I20260620 14:26:36.766747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34023
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:26:36.870065 9602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:36.870271 9602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:36.870303 9602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:36.874181 9602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:36.874222 9602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:36.874255 9602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:36.874270 9602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:36.874284 9602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:36.878600 9602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34023
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:36.879443 9602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:36.880589 9602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:36.888237 9607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:36.888248 9610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:36.888236 9608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:36.888281 9602 server_base.cc:1034] running on GCE node
I20260620 14:26:36.888885 9602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:36.889876 9602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:36.891054 9602 hybrid_clock.cc:648] HybridClock initialized: now 1781965596891043 us; error 46 us; skew 500 ppm
I20260620 14:26:36.892627 9602 webserver.cc:460] Webserver started at http://127.0.98.62:34099/ using document root <none> and password file <none>
I20260620 14:26:36.893131 9602 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:36.893239 9602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:36.893450 9602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:36.895226 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "cc293ab41eae43878340b5e973361ccf"
format_stamp: "Formatted at 2026-06-20 14:26:36 on dist-test-slave-pk3j"
server_key: "bb042f1ed4c611250373b425fd8b1406"
server_key_iv: "b846630917ab7d747cd8ca233dc95785"
server_key_version: "encryptionkey@0"
I20260620 14:26:36.895743 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc293ab41eae43878340b5e973361ccf"
format_stamp: "Formatted at 2026-06-20 14:26:36 on dist-test-slave-pk3j"
server_key: "bb042f1ed4c611250373b425fd8b1406"
server_key_iv: "b846630917ab7d747cd8ca233dc95785"
server_key_version: "encryptionkey@0"
I20260620 14:26:36.899034 9602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:36.901108 9616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:36.902204 9602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:36.902334 9602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "cc293ab41eae43878340b5e973361ccf"
format_stamp: "Formatted at 2026-06-20 14:26:36 on dist-test-slave-pk3j"
server_key: "bb042f1ed4c611250373b425fd8b1406"
server_key_iv: "b846630917ab7d747cd8ca233dc95785"
server_key_version: "encryptionkey@0"
I20260620 14:26:36.902436 9602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:36.920727 9602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:36.921325 9602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:36.921541 9602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:36.927556 9602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34023
I20260620 14:26:36.927577 9668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34023 every 8 connection(s)
I20260620 14:26:36.928414 9602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:36.931223 9669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:36.931794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 9602
I20260620 14:26:36.931885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:36.932188 392 external_mini_cluster.cc:1428] Setting key 912e0534feec3b0f29599e0fd7a13e2c
I20260620 14:26:36.937533 9669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf: Bootstrap starting.
I20260620 14:26:36.939831 9669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:36.940493 9669 log.cc:824] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:36.941711 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:36.933513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.0.98.62:34023)
0620 14:26:36.934117 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:36.934136 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:36.934178 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:26:36.934966 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:36.934989 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:36.935446 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:36.935783 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:36.936166 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:36.937623 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:36.938468 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:36.939116 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:36.939275 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:36.940013 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:36.940019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:36.940039 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:36.940382 (+ 343us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:36.940431 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:36.940449 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:36.940554 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:36.940713 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:26:36.940958 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260620 14:26:36.942267 9669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf: No bootstrap required, opened a new log
I20260620 14:26:36.945402 9669 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc293ab41eae43878340b5e973361ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } }
I20260620 14:26:36.945623 9669 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:36.945699 9669 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc293ab41eae43878340b5e973361ccf, State: Initialized, Role: FOLLOWER
I20260620 14:26:36.946087 9669 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc293ab41eae43878340b5e973361ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } }
I20260620 14:26:36.946226 9669 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:36.946310 9669 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:36.946424 9669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:36.947235 9669 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc293ab41eae43878340b5e973361ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } }
I20260620 14:26:36.947538 9669 leader_election.cc:302] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc293ab41eae43878340b5e973361ccf; no voters:
I20260620 14:26:36.947844 9669 leader_election.cc:288] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:36.947955 9674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:36.948189 9674 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [term 1 LEADER]: Becoming Leader. State: Replica: cc293ab41eae43878340b5e973361ccf, State: Running, Role: LEADER
I20260620 14:26:36.948522 9674 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc293ab41eae43878340b5e973361ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } }
I20260620 14:26:36.948925 9669 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:36.950004 9675 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc293ab41eae43878340b5e973361ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc293ab41eae43878340b5e973361ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } } }
I20260620 14:26:36.950126 9675 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:36.950182 9676 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc293ab41eae43878340b5e973361ccf. Latest consensus state: current_term: 1 leader_uuid: "cc293ab41eae43878340b5e973361ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc293ab41eae43878340b5e973361ccf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } } }
I20260620 14:26:36.950294 9676 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:36.950465 9683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:36.953433 9683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:36.958323 9683 catalog_manager.cc:1347] Generated new cluster ID: 5973fcefa6e8466b9d44314ed16661f7
I20260620 14:26:36.958393 9683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:36.989480 9683 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:36.990233 9683 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:36.993863 9683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc293ab41eae43878340b5e973361ccf: Generated new TSK 0
I20260620 14:26:36.994509 9683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:26:37.006275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--builtin_ntp_servers=127.0.98.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:26:37.111410 9693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:37.111651 9693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:37.111722 9693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:37.115622 9693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:37.115688 9693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:37.115765 9693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:37.120218 9693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:37.121125 9693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:37.122314 9693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:37.130738 9699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:37.130779 9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:37.130753 9698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:37.131064 9693 server_base.cc:1034] running on GCE node
I20260620 14:26:37.131395 9693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:37.131922 9693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:37.133100 9693 hybrid_clock.cc:648] HybridClock initialized: now 1781965597133083 us; error 50 us; skew 500 ppm
I20260620 14:26:37.134721 9693 webserver.cc:460] Webserver started at http://127.0.98.1:38905/ using document root <none> and password file <none>
I20260620 14:26:37.135229 9693 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:37.135316 9693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:37.135524 9693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:37.137347 9693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "939f275806c44932bd200400d36ae922"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "ae31dbe0be140ffe08d3f495e1c253db"
server_key_iv: "7b43d400ad903bb943a3a9befd05e163"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.137863 9693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "939f275806c44932bd200400d36ae922"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "ae31dbe0be140ffe08d3f495e1c253db"
server_key_iv: "7b43d400ad903bb943a3a9befd05e163"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.141049 9693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:26:37.143215 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.144286 9693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:37.144409 9693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "939f275806c44932bd200400d36ae922"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "ae31dbe0be140ffe08d3f495e1c253db"
server_key_iv: "7b43d400ad903bb943a3a9befd05e163"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.144517 9693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:37.154250 9693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:37.154779 9693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:37.154953 9693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:37.155438 9693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:37.156229 9693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:37.156306 9693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.156373 9693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:37.156407 9693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.164304 9693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42925
I20260620 14:26:37.164328 9820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42925 every 8 connection(s)
I20260620 14:26:37.165042 9693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:26:37.171895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 9693
I20260620 14:26:37.171983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:26:37.172295 392 external_mini_cluster.cc:1428] Setting key 841bf1ca943e25d422f9debfcbe879f1
I20260620 14:26:37.173193 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:37.166669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46075 (local address 127.0.98.62:34023)
0620 14:26:37.166793 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:37.166797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:37.167706 (+ 909us) server_negotiation.cc:407] Connection header received
0620 14:26:37.168917 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:37.168920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:37.168967 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:37.169045 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:37.169811 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.170341 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.171349 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.171559 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.171637 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:37.172659 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:37.172666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:37.172669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:37.172683 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:37.172712 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:37.172722 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:37.172811 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:37.173005 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:26:37.173058 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:26:37.174091 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:37.167003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34023 (local address 127.0.98.1:46075)
0620 14:26:37.167565 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:37.167599 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:37.168719 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:37.169179 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:37.169185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:37.169274 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:37.169680 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:37.169692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.170440 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.170448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:37.171245 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:37.171253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.171672 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.171675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:37.171797 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:37.172426 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:37.172427 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:37.172467 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:37.172518 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:37.172814 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:37.172826 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:37.172834 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:26:37.172963 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:26:37.173226 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
I20260620 14:26:37.174314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--builtin_ntp_servers=127.0.98.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:26:37.175282 9821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34023
I20260620 14:26:37.175518 9821 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:37.176050 9821 heartbeater.cc:511] Master 127.0.98.62:34023 requested a full tablet report, sending...
I20260620 14:26:37.177707 9632 ts_manager.cc:194] Registered new tserver with Master: 939f275806c44932bd200400d36ae922 (127.0.98.1:42925)
I20260620 14:26:37.178758 9632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46075
W20260620 14:26:37.280488 9824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:37.280808 9824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:37.280877 9824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:37.285022 9824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:37.285100 9824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:37.285188 9824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:37.290279 9824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:37.291357 9824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:37.292685 9824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:37.302380 9832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:37.302392 9829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:37.302398 9830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:37.302532 9824 server_base.cc:1034] running on GCE node
I20260620 14:26:37.303160 9824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:37.303864 9824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:37.305102 9824 hybrid_clock.cc:648] HybridClock initialized: now 1781965597305082 us; error 37 us; skew 500 ppm
I20260620 14:26:37.307122 9824 webserver.cc:460] Webserver started at http://127.0.98.2:38901/ using document root <none> and password file <none>
I20260620 14:26:37.307683 9824 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:37.307744 9824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:37.307911 9824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:37.310149 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "096c96f5da75479a9caeda89b97f2cef"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "72557bc5b637fdc0422b2b0dd299c55f"
server_key_iv: "66e6d3c337106e85d09c98591c691e88"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.310668 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "096c96f5da75479a9caeda89b97f2cef"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "72557bc5b637fdc0422b2b0dd299c55f"
server_key_iv: "66e6d3c337106e85d09c98591c691e88"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.314388 9824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:26:37.316880 9838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.318215 9824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:26:37.318353 9824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "096c96f5da75479a9caeda89b97f2cef"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "72557bc5b637fdc0422b2b0dd299c55f"
server_key_iv: "66e6d3c337106e85d09c98591c691e88"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.318470 9824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:37.335645 9824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:37.336263 9824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:37.336443 9824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:37.337006 9824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:37.337971 9824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:37.338017 9824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.338055 9824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:37.338112 9824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.346534 9824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46497
I20260620 14:26:37.346557 9951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46497 every 8 connection(s)
I20260620 14:26:37.347363 9824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:37.350750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 9824
I20260620 14:26:37.350874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:26:37.351184 392 external_mini_cluster.cc:1428] Setting key 587f51ef9c1dd7ea68010127f8b3ef75
I20260620 14:26:37.353328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--builtin_ntp_servers=127.0.98.20:36085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:26:37.357254 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:37.349099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52723 (local address 127.0.98.62:34023)
0620 14:26:37.349282 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:37.349286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:37.350341 (+ 1055us) server_negotiation.cc:407] Connection header received
0620 14:26:37.352128 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:37.352131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:37.352172 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:37.352231 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:37.353293 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.353867 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.355242 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.355474 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.355583 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:37.356660 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:37.356667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:37.356672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:37.356689 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:37.356716 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:37.356730 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:37.356809 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:37.356959 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 14:26:37.357029 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:26:37.357889 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:37.349543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34023 (local address 127.0.98.2:52723)
0620 14:26:37.350189 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:37.350226 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:37.351877 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:37.352421 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:37.352431 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:37.352531 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:37.353115 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:37.353137 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.353988 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.353995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:37.355115 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:37.355124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.355613 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.355617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:37.355774 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:37.356363 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:37.356364 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:37.356405 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:37.356462 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:37.356823 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:37.356829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:37.356833 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:26:37.356920 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:26:37.357107 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
I20260620 14:26:37.358958 9952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34023
I20260620 14:26:37.359153 9952 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:37.359601 9952 heartbeater.cc:511] Master 127.0.98.62:34023 requested a full tablet report, sending...
I20260620 14:26:37.360513 9632 ts_manager.cc:194] Registered new tserver with Master: 096c96f5da75479a9caeda89b97f2cef (127.0.98.2:46497)
I20260620 14:26:37.361078 9632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52723
W20260620 14:26:37.456344 9955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:37.456586 9955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:37.456656 9955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:37.460614 9955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:37.460682 9955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:37.460758 9955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:37.465178 9955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:37.470192 9955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:37.471863 9955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:37.480484 9961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:37.480561 9963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:37.480475 9960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:37.480773 9955 server_base.cc:1034] running on GCE node
I20260620 14:26:37.481130 9955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:37.481678 9955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:37.482856 9955 hybrid_clock.cc:648] HybridClock initialized: now 1781965597482830 us; error 49 us; skew 500 ppm
I20260620 14:26:37.484441 9955 webserver.cc:460] Webserver started at http://127.0.98.3:40415/ using document root <none> and password file <none>
I20260620 14:26:37.484951 9955 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:37.485035 9955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:37.485306 9955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:37.487135 9955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "7117ab5e48414c6c8bf3114899531f0c"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "3c4d7c55e0f6b057e4f003927cb9b525"
server_key_iv: "9d89b2d189eeb781052e2865d1fd470c"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.487689 9955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7117ab5e48414c6c8bf3114899531f0c"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "3c4d7c55e0f6b057e4f003927cb9b525"
server_key_iv: "9d89b2d189eeb781052e2865d1fd470c"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.491540 9955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:26:37.494089 9969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.495652 9955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:26:37.495774 9955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "7117ab5e48414c6c8bf3114899531f0c"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "3c4d7c55e0f6b057e4f003927cb9b525"
server_key_iv: "9d89b2d189eeb781052e2865d1fd470c"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.495910 9955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:37.544157 9955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:37.545009 9955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:37.545218 9955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:37.545955 9955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:37.547328 9955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:37.547394 9955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.547454 9955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:37.547503 9955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.560078 9955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44323
I20260620 14:26:37.560088 10082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44323 every 8 connection(s)
I20260620 14:26:37.561331 9955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:37.569674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 9955
I20260620 14:26:37.569789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:26:37.570115 392 external_mini_cluster.cc:1428] Setting key 1667567fcadc9a7dceda29b856939f0f
I20260620 14:26:37.570978 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:37.563096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36535 (local address 127.0.98.62:34023)
0620 14:26:37.563262 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:37.563269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:37.564243 (+ 974us) server_negotiation.cc:407] Connection header received
0620 14:26:37.565628 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:37.565634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:37.565700 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:37.565827 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:37.566633 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.567584 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.568664 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.569115 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.569272 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:37.570335 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:37.570339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:37.570343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:37.570359 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:37.570376 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:37.570384 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:37.570469 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:37.570728 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:26:37.570789 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:26:37.572026 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:37.563444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34023 (local address 127.0.98.3:36535)
0620 14:26:37.564089 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:37.564137 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:37.565430 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:37.565978 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:37.565983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:37.566061 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:37.566506 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:37.566517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.567716 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.567718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:37.568542 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:37.568551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:37.569291 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:37.569294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:37.569436 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:37.570033 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:37.570035 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:37.570071 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:37.570127 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:37.570490 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:37.570501 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:37.570511 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:26:37.570709 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 14:26:37.571029 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":116,"threads_started":1}
I20260620 14:26:37.573036 10083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34023
I20260620 14:26:37.573297 10083 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:37.573889 10083 heartbeater.cc:511] Master 127.0.98.62:34023 requested a full tablet report, sending...
I20260620 14:26:37.574890 9632 ts_manager.cc:194] Registered new tserver with Master: 7117ab5e48414c6c8bf3114899531f0c (127.0.98.3:44323)
I20260620 14:26:37.575415 9632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36535
I20260620 14:26:37.583568 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:26:37.586318 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:37.585450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.0.98.62:34023)
0620 14:26:37.585587 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:37.585594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:37.585715 (+ 121us) server_negotiation.cc:407] Connection header received
0620 14:26:37.585795 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:37.585799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:37.585852 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:37.585951 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:37.586175 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45782: BlockingRecv error: recv got EOF from 127.0.0.1:45782 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:26:37.586349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 9693
I20260620 14:26:37.592445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 9824
I20260620 14:26:37.597854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 9955
I20260620 14:26:37.602754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 9602
2026-06-20T14:26:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (865 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:26:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:37Z Disabled control of system clock
I20260620 14:26:37.630666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33035
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:26:37.732450 10094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:37.732650 10094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:37.732684 10094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:37.736474 10094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:37.736519 10094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:37.736536 10094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:37.736550 10094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:37.736564 10094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:37.741012 10094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33035
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:37.741828 10094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:37.742928 10094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:37.750211 10100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:37.750254 10099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:37.750209 10102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:37.750432 10094 server_base.cc:1034] running on GCE node
I20260620 14:26:37.751006 10094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:37.751936 10094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:37.753121 10094 hybrid_clock.cc:648] HybridClock initialized: now 1781965597753105 us; error 28 us; skew 500 ppm
I20260620 14:26:37.754830 10094 webserver.cc:460] Webserver started at http://127.0.98.62:39851/ using document root <none> and password file <none>
I20260620 14:26:37.755357 10094 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:37.755439 10094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:37.755640 10094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:37.757478 10094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "08f4f1777f624e50b06b42ac69e93675"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "1bc62b71978a9dcd3a39a9502d3019bd"
server_key_iv: "85b0f7994497cc7f2ce00cbea4ddf4ff"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.757974 10094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "08f4f1777f624e50b06b42ac69e93675"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "1bc62b71978a9dcd3a39a9502d3019bd"
server_key_iv: "85b0f7994497cc7f2ce00cbea4ddf4ff"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.761188 10094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:26:37.763422 10108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.764489 10094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:26:37.764613 10094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "08f4f1777f624e50b06b42ac69e93675"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "1bc62b71978a9dcd3a39a9502d3019bd"
server_key_iv: "85b0f7994497cc7f2ce00cbea4ddf4ff"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.764710 10094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:37.768705 10094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:37.769281 10094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:37.769439 10094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:37.775620 10094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33035
I20260620 14:26:37.775650 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33035 every 8 connection(s)
I20260620 14:26:37.776397 10094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:37.779165 10161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:37.783881 10161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675: Bootstrap starting.
I20260620 14:26:37.785478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 10094
I20260620 14:26:37.785567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:37.785840 392 external_mini_cluster.cc:1428] Setting key 31ec015bbda0b7e71013837a071a3397
I20260620 14:26:37.786249 10161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:37.787052 10161 log.cc:824] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:37.788822 10161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675: No bootstrap required, opened a new log
I20260620 14:26:37.790810 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:37.786860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.0.98.62:33035)
0620 14:26:37.787549 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:37.787569 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:37.787609 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:26:37.788260 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:37.788276 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:37.788529 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:37.788784 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:37.789665 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:37.789669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:37.789679 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:37.789688 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:37.789735 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:37.789749 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:37.789829 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:37.789972 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:26:37.790212 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":146,"threads_started":1}
I20260620 14:26:37.792695 10161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f4f1777f624e50b06b42ac69e93675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } }
I20260620 14:26:37.792961 10161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:37.793035 10161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f4f1777f624e50b06b42ac69e93675, State: Initialized, Role: FOLLOWER
I20260620 14:26:37.793428 10161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f4f1777f624e50b06b42ac69e93675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } }
I20260620 14:26:37.793543 10161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:37.793587 10161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:37.793650 10161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:37.794391 10161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f4f1777f624e50b06b42ac69e93675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } }
I20260620 14:26:37.794648 10161 leader_election.cc:302] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08f4f1777f624e50b06b42ac69e93675; no voters:
I20260620 14:26:37.794869 10161 leader_election.cc:288] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:37.794945 10166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:37.795209 10166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [term 1 LEADER]: Becoming Leader. State: Replica: 08f4f1777f624e50b06b42ac69e93675, State: Running, Role: LEADER
I20260620 14:26:37.795471 10166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f4f1777f624e50b06b42ac69e93675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } }
I20260620 14:26:37.795953 10161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:37.797096 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08f4f1777f624e50b06b42ac69e93675. Latest consensus state: current_term: 1 leader_uuid: "08f4f1777f624e50b06b42ac69e93675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f4f1777f624e50b06b42ac69e93675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } } }
I20260620 14:26:37.797245 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:37.797456 10167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08f4f1777f624e50b06b42ac69e93675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f4f1777f624e50b06b42ac69e93675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33035 } } }
I20260620 14:26:37.797540 10167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:37.798460 10175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:37.800904 10175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:37.805564 10175 catalog_manager.cc:1347] Generated new cluster ID: a7dc1a6d700d410aa2ea4738821003b5
I20260620 14:26:37.805624 10175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:37.816111 10175 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:37.816375 10175 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:37.824507 10175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08f4f1777f624e50b06b42ac69e93675: Generated new TSK 0
I20260620 14:26:37.825016 10175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:26:37.837100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33035
--builtin_ntp_servers=127.0.98.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:26:37.948555 10185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:37.948772 10185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:37.948843 10185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:37.952782 10185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:37.952831 10185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:37.952929 10185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:37.957556 10185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:37.958501 10185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:37.959764 10185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:37.968641 10193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:37.968649 10190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:37.968648 10191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:37.968819 10185 server_base.cc:1034] running on GCE node
I20260620 14:26:37.969329 10185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:37.969852 10185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:37.971019 10185 hybrid_clock.cc:648] HybridClock initialized: now 1781965597971009 us; error 39 us; skew 500 ppm
I20260620 14:26:37.972571 10185 webserver.cc:460] Webserver started at http://127.0.98.1:44703/ using document root <none> and password file <none>
I20260620 14:26:37.973032 10185 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:37.973085 10185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:37.973310 10185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:37.975095 10185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "57dc82a43eaf4b7e843aca3f7987feb6"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "cbbc1366e1c9522827ce5edd02ada605"
server_key_iv: "7637e0ae40184187f050d2c534aff257"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.975548 10185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "57dc82a43eaf4b7e843aca3f7987feb6"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "cbbc1366e1c9522827ce5edd02ada605"
server_key_iv: "7637e0ae40184187f050d2c534aff257"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.978744 10185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:26:37.980836 10199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.982002 10185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:26:37.982136 10185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "57dc82a43eaf4b7e843aca3f7987feb6"
format_stamp: "Formatted at 2026-06-20 14:26:37 on dist-test-slave-pk3j"
server_key: "cbbc1366e1c9522827ce5edd02ada605"
server_key_iv: "7637e0ae40184187f050d2c534aff257"
server_key_version: "encryptionkey@0"
I20260620 14:26:37.982228 10185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:37.986266 10185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:37.986775 10185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:37.986943 10185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:37.987448 10185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:37.988265 10185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:37.988308 10185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.988372 10185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:37.988404 10185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:37.997102 10185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37007
I20260620 14:26:37.997145 10312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37007 every 8 connection(s)
I20260620 14:26:37.997937 10185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:26:38.002439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 10185
I20260620 14:26:38.002530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:26:38.002856 392 external_mini_cluster.cc:1428] Setting key e196394ccbe378020de474f728878c2f
I20260620 14:26:38.003690 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:37.999584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32815 (local address 127.0.98.62:33035)
0620 14:26:37.999708 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:37.999712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:38.000613 (+ 901us) server_negotiation.cc:407] Connection header received
0620 14:26:38.001951 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:38.001955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:38.001997 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:38.002111 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:38.003189 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:38.003195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:38.003198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:38.003210 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:38.003231 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:38.003241 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:38.003331 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:38.003531 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 14:26:38.003571 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:26:38.004679 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:37.999914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33035 (local address 127.0.98.1:32815)
0620 14:26:38.000485 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:38.000519 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:38.001731 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:38.002252 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:38.002258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:38.002337 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:38.002935 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:38.002938 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:38.002987 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:38.003048 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:38.003352 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:38.003363 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:38.003369 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:26:38.003544 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 14:26:38.003722 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
I20260620 14:26:38.004853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33035
--builtin_ntp_servers=127.0.98.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:26:38.005995 10313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33035
I20260620 14:26:38.006237 10313 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:38.006677 10313 heartbeater.cc:511] Master 127.0.98.62:33035 requested a full tablet report, sending...
I20260620 14:26:38.008077 10125 ts_manager.cc:194] Registered new tserver with Master: 57dc82a43eaf4b7e843aca3f7987feb6 (127.0.98.1:37007)
W20260620 14:26:38.108273 10316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:38.108515 10316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:38.108585 10316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:38.112586 10316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:38.112658 10316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:38.112761 10316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:38.117355 10316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:38.118234 10316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:38.119431 10316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:38.128304 10324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:38.128301 10321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:38.128324 10322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:38.128650 10316 server_base.cc:1034] running on GCE node
I20260620 14:26:38.128998 10316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:38.129599 10316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:38.130767 10316 hybrid_clock.cc:648] HybridClock initialized: now 1781965598130737 us; error 42 us; skew 500 ppm
I20260620 14:26:38.132367 10316 webserver.cc:460] Webserver started at http://127.0.98.2:45693/ using document root <none> and password file <none>
I20260620 14:26:38.132872 10316 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:38.132928 10316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:38.133154 10316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:38.135521 10316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ecef75c90d74091aaf529455387aed3"
format_stamp: "Formatted at 2026-06-20 14:26:38 on dist-test-slave-pk3j"
server_key: "380a0ceb2da9d2beb262d2583cab5bcc"
server_key_iv: "f0155ac2a8bf4d02a88eff446b3508a5"
server_key_version: "encryptionkey@0"
I20260620 14:26:38.136229 10316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ecef75c90d74091aaf529455387aed3"
format_stamp: "Formatted at 2026-06-20 14:26:38 on dist-test-slave-pk3j"
server_key: "380a0ceb2da9d2beb262d2583cab5bcc"
server_key_iv: "f0155ac2a8bf4d02a88eff446b3508a5"
server_key_version: "encryptionkey@0"
I20260620 14:26:38.140882 10316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 14:26:38.143030 10330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:38.144181 10316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:26:38.144306 10316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "0ecef75c90d74091aaf529455387aed3"
format_stamp: "Formatted at 2026-06-20 14:26:38 on dist-test-slave-pk3j"
server_key: "380a0ceb2da9d2beb262d2583cab5bcc"
server_key_iv: "f0155ac2a8bf4d02a88eff446b3508a5"
server_key_version: "encryptionkey@0"
I20260620 14:26:38.144408 10316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:38.148587 10316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:38.149106 10316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:38.149299 10316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:38.149781 10316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:38.150564 10316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:38.150640 10316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:38.150761 10316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:38.150825 10316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:38.158736 10316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43671
I20260620 14:26:38.158762 10443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43671 every 8 connection(s)
I20260620 14:26:38.159474 10316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:38.160379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 10316
I20260620 14:26:38.160507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:26:38.160775 392 external_mini_cluster.cc:1428] Setting key 122026c10783f8949848f872168171e6
I20260620 14:26:38.162681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33035
--builtin_ntp_servers=127.0.98.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:26:38.165395 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:38.161180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53913 (local address 127.0.98.62:33035)
0620 14:26:38.161334 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:38.161338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:38.162418 (+ 1080us) server_negotiation.cc:407] Connection header received
0620 14:26:38.163747 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:38.163750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:38.163789 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:38.163863 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:38.164875 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:38.164881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:38.164885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:38.164899 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:38.164922 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:38.164931 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:38.165022 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:38.165190 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:26:38.165271 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:26:38.166097 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:38.161600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33035 (local address 127.0.98.2:53913)
0620 14:26:38.162294 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:38.162329 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:38.163541 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:38.164005 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:38.164011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:38.164089 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:38.164651 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:38.164653 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:38.164685 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:38.164742 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:38.165032 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:38.165039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:38.165044 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:26:38.165253 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 14:26:38.165459 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
I20260620 14:26:38.167126 10444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33035
I20260620 14:26:38.167354 10444 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:38.167817 10444 heartbeater.cc:511] Master 127.0.98.62:33035 requested a full tablet report, sending...
I20260620 14:26:38.168923 10125 ts_manager.cc:194] Registered new tserver with Master: 0ecef75c90d74091aaf529455387aed3 (127.0.98.2:43671)
W20260620 14:26:38.266177 10447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:38.266397 10447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:38.266446 10447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:38.270401 10447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:38.270442 10447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:38.270522 10447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:38.275167 10447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:38.276038 10447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:38.277189 10447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:38.285447 10452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:38.285466 10453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:38.285466 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:38.285652 10447 server_base.cc:1034] running on GCE node
I20260620 14:26:38.286111 10447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:38.286636 10447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:38.287783 10447 hybrid_clock.cc:648] HybridClock initialized: now 1781965598287771 us; error 22 us; skew 500 ppm
I20260620 14:26:38.289405 10447 webserver.cc:460] Webserver started at http://127.0.98.3:35537/ using document root <none> and password file <none>
I20260620 14:26:38.289871 10447 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:38.289925 10447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:38.290067 10447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:38.291915 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "7e3b0455b86d4e3a88ea19581b87b212"
format_stamp: "Formatted at 2026-06-20 14:26:38 on dist-test-slave-pk3j"
server_key: "d62a1f75c00700181733ae047f2f0e49"
server_key_iv: "1c38b0c28226f92b3a3336e5dc4e583d"
server_key_version: "encryptionkey@0"
I20260620 14:26:38.292367 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7e3b0455b86d4e3a88ea19581b87b212"
format_stamp: "Formatted at 2026-06-20 14:26:38 on dist-test-slave-pk3j"
server_key: "d62a1f75c00700181733ae047f2f0e49"
server_key_iv: "1c38b0c28226f92b3a3336e5dc4e583d"
server_key_version: "encryptionkey@0"
I20260620 14:26:38.295509 10447 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:26:38.297981 10461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:38.299152 10447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:38.299237 10447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "7e3b0455b86d4e3a88ea19581b87b212"
format_stamp: "Formatted at 2026-06-20 14:26:38 on dist-test-slave-pk3j"
server_key: "d62a1f75c00700181733ae047f2f0e49"
server_key_iv: "1c38b0c28226f92b3a3336e5dc4e583d"
server_key_version: "encryptionkey@0"
I20260620 14:26:38.299324 10447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:38.303341 10447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:38.303815 10447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:38.303936 10447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:38.304345 10447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:38.305104 10447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:38.305148 10447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:38.305182 10447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:38.305254 10447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:38.313306 10447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39939
I20260620 14:26:38.313364 10574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39939 every 8 connection(s)
I20260620 14:26:38.314043 10447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:38.318487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 10447
I20260620 14:26:38.318601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:26:38.318846 392 external_mini_cluster.cc:1428] Setting key fc00355fea2d2a323d19842e55052463
I20260620 14:26:38.319466 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:38.315624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52479 (local address 127.0.98.62:33035)
0620 14:26:38.315743 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:38.315747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:38.316663 (+ 916us) server_negotiation.cc:407] Connection header received
0620 14:26:38.317825 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:38.317827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:38.317862 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:38.317930 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:38.318989 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:38.318993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:38.318995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:26:38.319007 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:38.319032 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:26:38.319040 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:38.319139 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:38.319277 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:26:38.319331 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:26:38.320374 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:38.315956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33035 (local address 127.0.98.3:52479)
0620 14:26:38.316539 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:38.316571 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:38.317660 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:38.318090 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:38.318096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:38.318175 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:38.318751 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:26:38.318753 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:38.318786 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:26:38.318845 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:38.319123 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:38.319130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:38.319135 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:26:38.319314 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 14:26:38.319514 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":105,"threads_started":1}
I20260620 14:26:38.321301 10575 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33035
I20260620 14:26:38.321496 10575 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:38.321835 10575 heartbeater.cc:511] Master 127.0.98.62:33035 requested a full tablet report, sending...
I20260620 14:26:38.322680 10125 ts_manager.cc:194] Registered new tserver with Master: 7e3b0455b86d4e3a88ea19581b87b212 (127.0.98.3:39939)
I20260620 14:26:38.332362 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:26:38.335044 10164 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:26:38.334158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.0.98.62:33035)
0620 14:26:38.334295 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:38.334301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:38.334496 (+ 195us) server_negotiation.cc:407] Connection header received
0620 14:26:38.334632 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:38.334634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:38.334672 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:38.334753 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:38.334948 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56416: BlockingRecv error: recv got EOF from 127.0.0.1:56416 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:26:38.335242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 10185
I20260620 14:26:38.340593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 10316
I20260620 14:26:38.345741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 10447
I20260620 14:26:38.350710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 10094
2026-06-20T14:26:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (747 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:26:38 dist-test-slave-pk3j krb5kdc[10585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:26:38 dist-test-slave-pk3j krb5kdc[10585](info): set up 2 sockets
Jun 20 14:26:38 dist-test-slave-pk3j krb5kdc[10585](info): commencing operation
krb5kdc: starting...
W20260620 14:26:39.369877 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:26:39 dist-test-slave-pk3j krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:26:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:26:39.476755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46855
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:39.577775 10598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:39.577991 10598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:39.578028 10598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:39.582053 10598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:26:39.582099 10598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:39.582139 10598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:39.582183 10598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:26:39.582209 10598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:26:39.586943 10598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:39.587796 10598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:39.588725 10598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:26:39.596621 10598 server_base.cc:1034] running on GCE node
W20260620 14:26:39.596580 10603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:39.596601 10606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:39.596810 10604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:39.597364 10598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:39.598285 10598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:39.599476 10598 hybrid_clock.cc:648] HybridClock initialized: now 1781965599599462 us; error 34 us; skew 500 ppm
Jun 20 14:26:39 dist-test-slave-pk3j krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:39.601989 10598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:26:39.602715 10598 webserver.cc:460] Webserver started at http://127.0.98.62:37191/ using document root <none> and password file <none>
I20260620 14:26:39.603220 10598 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:39.603302 10598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:39.603510 10598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:39.605341 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "432cf4bb003d4bad8a665fe448dc2314"
format_stamp: "Formatted at 2026-06-20 14:26:39 on dist-test-slave-pk3j"
server_key: "2d10a87230ae92472c997d0c4af5a0b8"
server_key_iv: "24e7f549eac7a53fd0953f25b2599d9b"
server_key_version: "encryptionkey@0"
I20260620 14:26:39.605839 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "432cf4bb003d4bad8a665fe448dc2314"
format_stamp: "Formatted at 2026-06-20 14:26:39 on dist-test-slave-pk3j"
server_key: "2d10a87230ae92472c997d0c4af5a0b8"
server_key_iv: "24e7f549eac7a53fd0953f25b2599d9b"
server_key_version: "encryptionkey@0"
I20260620 14:26:39.609081 10598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:26:39.611253 10613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:39.612334 10598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:39.612484 10598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "432cf4bb003d4bad8a665fe448dc2314"
format_stamp: "Formatted at 2026-06-20 14:26:39 on dist-test-slave-pk3j"
server_key: "2d10a87230ae92472c997d0c4af5a0b8"
server_key_iv: "24e7f549eac7a53fd0953f25b2599d9b"
server_key_version: "encryptionkey@0"
I20260620 14:26:39.612582 10598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:39.629019 10598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:39.631524 10598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:39.631702 10598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:39.637904 10598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46855
I20260620 14:26:39.637914 10665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46855 every 8 connection(s)
I20260620 14:26:39.638724 10598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:26:39.641103 10666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:39.641578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 10598
I20260620 14:26:39.641688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:26:39.641963 392 external_mini_cluster.cc:1428] Setting key 073a82581a84b86d06b3572660df8a92
I20260620 14:26:39.647353 10666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314: Bootstrap starting.
I20260620 14:26:39.649863 10666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314: Neither blocks nor log segments found. Creating new log.
Jun 20 14:26:39 dist-test-slave-pk3j krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:39.650557 10666 log.cc:824] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:39.652191 10666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314: No bootstrap required, opened a new log
I20260620 14:26:39.654870 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:39.643097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.0.98.62:46855)
0620 14:26:39.643431 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:39.643444 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:39.643474 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:26:39.644040 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:39.644054 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:39.644339 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:39.644588 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:39.645428 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:39.646333 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:39.647541 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:39.647969 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:39.648079 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:39.650677 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:39.650681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:39.650692 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:39.650714 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:39.652402 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:39.652807 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:39.652816 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:39.652820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:39.652922 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:39.653236 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:39.653240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:39.653241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:39.653572 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:39.653768 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:39.654040 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:26:39.654280 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
I20260620 14:26:39.655547 10666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432cf4bb003d4bad8a665fe448dc2314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46855 } }
I20260620 14:26:39.655755 10666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:39.655807 10666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432cf4bb003d4bad8a665fe448dc2314, State: Initialized, Role: FOLLOWER
I20260620 14:26:39.656234 10666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432cf4bb003d4bad8a665fe448dc2314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46855 } }
I20260620 14:26:39.656375 10666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:26:39.656455 10666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:26:39.656566 10666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:39.657341 10666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432cf4bb003d4bad8a665fe448dc2314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46855 } }
I20260620 14:26:39.657639 10666 leader_election.cc:302] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 432cf4bb003d4bad8a665fe448dc2314; no voters:
I20260620 14:26:39.657943 10666 leader_election.cc:288] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:26:39.658035 10671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:39.658272 10671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [term 1 LEADER]: Becoming Leader. State: Replica: 432cf4bb003d4bad8a665fe448dc2314, State: Running, Role: LEADER
I20260620 14:26:39.658563 10671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432cf4bb003d4bad8a665fe448dc2314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46855 } }
I20260620 14:26:39.659017 10666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:26:39.660084 10672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "432cf4bb003d4bad8a665fe448dc2314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432cf4bb003d4bad8a665fe448dc2314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46855 } } }
I20260620 14:26:39.660218 10672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:39.660076 10673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 432cf4bb003d4bad8a665fe448dc2314. Latest consensus state: current_term: 1 leader_uuid: "432cf4bb003d4bad8a665fe448dc2314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "432cf4bb003d4bad8a665fe448dc2314" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46855 } } }
I20260620 14:26:39.660352 10673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314 [sys.catalog]: This master's current role is: LEADER
I20260620 14:26:39.660624 10680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:26:39.663426 10680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:26:39.668121 10680 catalog_manager.cc:1347] Generated new cluster ID: ac1a1dcc4f37439f97ca53b2db6f59f0
I20260620 14:26:39.668205 10680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:26:39.674791 10680 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:26:39.675622 10680 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:26:39.684454 10680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314: Generated new TSK 0
I20260620 14:26:39.684981 10680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:26:39.712234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46855
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:39.815718 10692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:39.815994 10692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:39.816092 10692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:39.819973 10692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:39.820046 10692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:39.820130 10692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:26:39.824724 10692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:39.825603 10692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:39.826473 10692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:26:39.834964 10692 server_base.cc:1034] running on GCE node
W20260620 14:26:39.834959 10697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:39.834982 10698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:39.834983 10700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:39.835531 10692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:39.836019 10692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:39.837231 10692 hybrid_clock.cc:648] HybridClock initialized: now 1781965599837184 us; error 33 us; skew 500 ppm
Jun 20 14:26:39 dist-test-slave-pk3j krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:39.839653 10692 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:26:39.840372 10692 webserver.cc:460] Webserver started at http://127.0.98.1:44081/ using document root <none> and password file <none>
I20260620 14:26:39.840857 10692 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:39.840941 10692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:39.841153 10692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:39.842907 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "12afe5ea4691474fa6b108dd5139d2a5"
format_stamp: "Formatted at 2026-06-20 14:26:39 on dist-test-slave-pk3j"
server_key: "02db71368fbd8d41bb9ce3a4bed611e6"
server_key_iv: "304a3fb6b62a7c45b7dcdfd8a016b66b"
server_key_version: "encryptionkey@0"
I20260620 14:26:39.843426 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "12afe5ea4691474fa6b108dd5139d2a5"
format_stamp: "Formatted at 2026-06-20 14:26:39 on dist-test-slave-pk3j"
server_key: "02db71368fbd8d41bb9ce3a4bed611e6"
server_key_iv: "304a3fb6b62a7c45b7dcdfd8a016b66b"
server_key_version: "encryptionkey@0"
I20260620 14:26:39.846776 10692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:39.848827 10707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:39.849916 10692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:39.850045 10692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "12afe5ea4691474fa6b108dd5139d2a5"
format_stamp: "Formatted at 2026-06-20 14:26:39 on dist-test-slave-pk3j"
server_key: "02db71368fbd8d41bb9ce3a4bed611e6"
server_key_iv: "304a3fb6b62a7c45b7dcdfd8a016b66b"
server_key_version: "encryptionkey@0"
I20260620 14:26:39.850143 10692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:39.864372 10692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:39.866720 10692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:39.866901 10692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:39.867401 10692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:39.868101 10692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:39.868171 10692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:39.868260 10692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:39.868304 10692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:39.876652 10692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38793
I20260620 14:26:39.876684 10820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38793 every 8 connection(s)
I20260620 14:26:39.877427 10692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:26:39 dist-test-slave-pk3j krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:39.887305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 10692
I20260620 14:26:39.887434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:26:39.887684 392 external_mini_cluster.cc:1428] Setting key 28f15b1ca597a76b91b6c98e94fc3bcc
I20260620 14:26:39.889248 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:39.879014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46071 (local address 127.0.98.62:46855)
0620 14:26:39.879149 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:39.879152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:39.880005 (+ 853us) server_negotiation.cc:407] Connection header received
0620 14:26:39.880819 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:39.880822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:39.880893 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:39.880991 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:39.882088 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:39.882589 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:39.883643 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:39.883861 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:39.883969 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:39.886076 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:39.886080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:39.886082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:39.886112 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:39.887259 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:39.887780 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:39.887784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:39.887786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:39.887866 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:39.888217 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:39.888221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:39.888223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:39.888391 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:39.888496 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:39.888961 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 14:26:39.889081 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:26:39.889935 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:39.879308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46855 (local address 127.0.98.1:46071)
0620 14:26:39.879875 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:39.879907 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:39.880630 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:39.881157 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:39.881163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:39.881553 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:39.881948 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:39.881960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:39.882706 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:39.882710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:39.883510 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:39.883518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:39.883994 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:39.883997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:39.884156 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:39.884594 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:39.884608 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:39.885929 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:39.887379 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:39.887384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:39.887395 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:39.887683 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:39.887953 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:39.887958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:39.887960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:39.888116 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:39.888494 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:39.888500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:39.888702 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 14:26:39.888893 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:26:39.889137 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":105,"threads_started":1}
I20260620 14:26:39.891074 10821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46855
I20260620 14:26:39.891276 10821 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:39.891709 10821 heartbeater.cc:511] Master 127.0.98.62:46855 requested a full tablet report, sending...
I20260620 14:26:39.893040 10630 ts_manager.cc:194] Registered new tserver with Master: 12afe5ea4691474fa6b108dd5139d2a5 (127.0.98.1:38793)
I20260620 14:26:39.894415 10630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46071
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:26:39.910161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46855
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:40.011646 10826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:40.011900 10826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:40.012008 10826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:40.015858 10826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:40.015924 10826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:40.016004 10826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:26:40.020865 10826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:40.021692 10826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:40.022527 10826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:40.030987 10831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:40.030973 10834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:40.030952 10832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:40.031096 10826 server_base.cc:1034] running on GCE node
I20260620 14:26:40.031553 10826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:40.032105 10826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:40.033273 10826 hybrid_clock.cc:648] HybridClock initialized: now 1781965600033242 us; error 41 us; skew 500 ppm
Jun 20 14:26:40 dist-test-slave-pk3j krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:40.035877 10826 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:26:40.036593 10826 webserver.cc:460] Webserver started at http://127.0.98.2:43523/ using document root <none> and password file <none>
I20260620 14:26:40.037021 10826 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:40.037072 10826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:40.037258 10826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:40.038996 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "ffa4bab034064a9ea69914868988a383"
format_stamp: "Formatted at 2026-06-20 14:26:40 on dist-test-slave-pk3j"
server_key: "063bf81cc105392893b890aecae765b7"
server_key_iv: "20cae1cb08a661f398b353970ea27df9"
server_key_version: "encryptionkey@0"
I20260620 14:26:40.039470 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ffa4bab034064a9ea69914868988a383"
format_stamp: "Formatted at 2026-06-20 14:26:40 on dist-test-slave-pk3j"
server_key: "063bf81cc105392893b890aecae765b7"
server_key_iv: "20cae1cb08a661f398b353970ea27df9"
server_key_version: "encryptionkey@0"
I20260620 14:26:40.042723 10826 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:26:40.044752 10841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:40.045892 10826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:26:40.045975 10826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "ffa4bab034064a9ea69914868988a383"
format_stamp: "Formatted at 2026-06-20 14:26:40 on dist-test-slave-pk3j"
server_key: "063bf81cc105392893b890aecae765b7"
server_key_iv: "20cae1cb08a661f398b353970ea27df9"
server_key_version: "encryptionkey@0"
I20260620 14:26:40.046046 10826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:40.054795 10826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:40.057128 10826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:40.057308 10826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:40.057762 10826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:40.058419 10826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:40.058468 10826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:40.058502 10826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:40.058523 10826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:40.066501 10826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35663
I20260620 14:26:40.066521 10954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35663 every 8 connection(s)
I20260620 14:26:40.067212 10826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:26:40.075088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 10826
I20260620 14:26:40.075206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
Jun 20 14:26:40 dist-test-slave-pk3j krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:40.075506 392 external_mini_cluster.cc:1428] Setting key 2c11d236eb2f1302b992ba84e0cd4f9d
I20260620 14:26:40.079702 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.068782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47859 (local address 127.0.98.62:46855)
0620 14:26:40.068921 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:40.068926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:40.069876 (+ 950us) server_negotiation.cc:407] Connection header received
0620 14:26:40.070619 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:40.070621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:40.070664 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:40.070733 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:40.071739 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.072234 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.073328 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.073544 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.073623 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.076012 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:40.076015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:40.076017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:40.076039 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:40.077545 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:40.078142 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:40.078144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:40.078146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:40.078217 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:40.078610 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:40.078616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:40.078619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:40.078847 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:40.078965 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:40.079373 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 14:26:40.079508 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:26:40.080442 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.069136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46855 (local address 127.0.98.2:47859)
0620 14:26:40.069745 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:40.069781 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:40.070447 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:40.070899 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:40.070905 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:40.071221 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:40.071622 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.071634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.072342 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.072345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.073126 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.073140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.073661 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.073664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.073788 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.074245 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:40.074260 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:40.075839 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:40.077656 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:40.077664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:40.077680 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:40.078032 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:40.078305 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:40.078311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:40.078316 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:40.078501 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:40.078949 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:40.078954 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:40.079153 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 14:26:40.079295 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 14:26:40.079514 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":101,"threads_started":1}
I20260620 14:26:40.081478 10955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46855
I20260620 14:26:40.081696 10955 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:40.082108 10955 heartbeater.cc:511] Master 127.0.98.62:46855 requested a full tablet report, sending...
I20260620 14:26:40.083076 10630 ts_manager.cc:194] Registered new tserver with Master: ffa4bab034064a9ea69914868988a383 (127.0.98.2:35663)
I20260620 14:26:40.083603 10630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47859
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:26:40.099521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46855
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:26:40.202927 10960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:26:40.203130 10960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:26:40.203167 10960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:26:40.206987 10960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:26:40.207034 10960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:26:40.207126 10960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:26:40.211978 10960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:26:40.212813 10960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:26:40.213696 10960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:26:40.222172 10965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:40.222138 10966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:26:40.222186 10968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:26:40.222491 10960 server_base.cc:1034] running on GCE node
I20260620 14:26:40.222805 10960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:26:40.223277 10960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:26:40.224423 10960 hybrid_clock.cc:648] HybridClock initialized: now 1781965600224414 us; error 23 us; skew 500 ppm
Jun 20 14:26:40 dist-test-slave-pk3j krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:26:40.226971 10960 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:26:40.227747 10960 webserver.cc:460] Webserver started at http://127.0.98.3:41463/ using document root <none> and password file <none>
I20260620 14:26:40.228269 10960 fs_manager.cc:359] Metadata directory not provided
I20260620 14:26:40.228358 10960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:26:40.228570 10960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:26:40.230393 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "dbf551fd81694a7899a0f2861fe3690b"
format_stamp: "Formatted at 2026-06-20 14:26:40 on dist-test-slave-pk3j"
server_key: "a5c9b5d130603d037936f18d606b3eae"
server_key_iv: "a20d54cfc8cc154e7263db82c0a8dae7"
server_key_version: "encryptionkey@0"
I20260620 14:26:40.230901 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dbf551fd81694a7899a0f2861fe3690b"
format_stamp: "Formatted at 2026-06-20 14:26:40 on dist-test-slave-pk3j"
server_key: "a5c9b5d130603d037936f18d606b3eae"
server_key_iv: "a20d54cfc8cc154e7263db82c0a8dae7"
server_key_version: "encryptionkey@0"
I20260620 14:26:40.234491 10960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:26:40.236527 10975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:40.237681 10960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:26:40.237812 10960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "dbf551fd81694a7899a0f2861fe3690b"
format_stamp: "Formatted at 2026-06-20 14:26:40 on dist-test-slave-pk3j"
server_key: "a5c9b5d130603d037936f18d606b3eae"
server_key_iv: "a20d54cfc8cc154e7263db82c0a8dae7"
server_key_version: "encryptionkey@0"
I20260620 14:26:40.237912 10960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:26:40.256356 10960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:26:40.259364 10960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:26:40.259575 10960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:26:40.260128 10960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:26:40.260919 10960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:26:40.260987 10960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:40.261056 10960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:26:40.261099 10960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:26:40.269192 10960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36885
I20260620 14:26:40.269265 11088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36885 every 8 connection(s)
I20260620 14:26:40.270056 10960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:26:40.274171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 10960
I20260620 14:26:40.274257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:26:40.274495 392 external_mini_cluster.cc:1428] Setting key 8fe39ffb1a4a1729531cdba74a411484
Jun 20 14:26:40 dist-test-slave-pk3j krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:26:40.282330 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.271620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56457 (local address 127.0.98.62:46855)
0620 14:26:40.271746 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:40.271751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:40.272655 (+ 904us) server_negotiation.cc:407] Connection header received
0620 14:26:40.273532 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:40.273536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:40.273595 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:40.273672 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:40.274798 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.275495 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.276654 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.276887 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.276993 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.279211 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:40.279215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:40.279218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:40.279243 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:40.280428 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:40.280898 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:40.280904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:40.280906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:40.280985 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:40.281373 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:40.281376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:40.281378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:40.281544 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:40.281646 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:40.282076 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 14:26:40.282180 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:26:40.282959 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.271929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46855 (local address 127.0.98.3:56457)
0620 14:26:40.272514 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:40.272549 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:40.273334 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:40.273823 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:40.273829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:40.274153 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:26:40.274662 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.274676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.275611 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.275614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.276541 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.276549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.277021 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.277025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.277176 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.277706 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:26:40.277720 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:26:40.279062 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:26:40.280540 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:40.280545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:40.280558 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:40.280791 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:40.281082 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:26:40.281085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:26:40.281087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:26:40.281262 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:26:40.281644 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:26:40.281650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:26:40.281861 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 14:26:40.282023 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:26:40.282288 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":119,"threads_started":1}
I20260620 14:26:40.283917 11089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46855
I20260620 14:26:40.284103 11089 heartbeater.cc:464] Registering TS with master...
I20260620 14:26:40.284487 11089 heartbeater.cc:511] Master 127.0.98.62:46855 requested a full tablet report, sending...
I20260620 14:26:40.285378 10630 ts_manager.cc:194] Registered new tserver with Master: dbf551fd81694a7899a0f2861fe3690b (127.0.98.3:36885)
I20260620 14:26:40.285910 10630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56457
I20260620 14:26:40.287858 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:26:40.295897 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.289580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.0.98.62:46855)
0620 14:26:40.289704 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:40.289708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:40.289849 (+ 141us) server_negotiation.cc:407] Connection header received
0620 14:26:40.289935 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:40.289937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:40.289969 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:40.290025 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:26:40.290573 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.291027 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.291786 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.291998 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.292067 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.293275 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:26:40.293278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:26:40.293279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:26:40.293296 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:26:40.294541 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:40.294888 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:40.294890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:40.294891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:40.294955 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:26:40.295193 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:26:40.295194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:26:40.295195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:26:40.295362 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:26:40.295454 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:40.295628 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:26:40.295702 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:26:40.298444 10630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:26:40.301174 10630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:26:40.314860 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.309698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38892 (local address 127.0.98.1:38793)
0620 14:26:40.310179 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:40.310195 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:40.310939 (+ 744us) server_negotiation.cc:407] Connection header received
0620 14:26:40.311012 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:40.311017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:40.311179 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:40.311320 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:40.311524 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.312438 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.313739 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.314221 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.314339 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.314463 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:40.314637 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:26:40.314719 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":127,"threads_started":1}
I20260620 14:26:40.314860 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.309520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38793 (local address 127.0.0.1:38892)
0620 14:26:40.310720 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:40.310735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:40.310813 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:40.311323 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:40.311326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:40.311343 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:40.311422 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.311427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.312578 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.312582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.313622 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.313631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.314338 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.314341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.314437 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.314469 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:26:40.314600 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:26:40.314710 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":81,"threads_started":1}
I20260620 14:26:40.315172 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.309488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.0.98.3:36885)
0620 14:26:40.310032 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:40.310043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:40.310065 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:26:40.310129 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:40.310133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:40.310321 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:40.310495 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:40.310775 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.311947 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.313481 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.314223 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.314326 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.314453 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:40.314931 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 14:26:40.315053 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":133,"threads_started":1}
I20260620 14:26:40.315127 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.308917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35663 (local address 127.0.0.1:52572)
0620 14:26:40.309359 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:40.309374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:40.309471 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:40.310646 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:40.310649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:40.310666 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:40.310778 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.310787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.311921 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.311927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.313369 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.313376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.314776 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.314779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.314866 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.314890 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:26:40.314957 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:26:40.315017 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":59,"thread_start_us":87,"threads_started":1}
I20260620 14:26:40.315496 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.309804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.0.98.2:35663)
0620 14:26:40.310222 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:40.310231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:40.310250 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:26:40.310342 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:40.310347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:40.310506 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:40.310655 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:40.310925 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.311794 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.313777 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.314520 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.314611 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.314725 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:40.315216 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 14:26:40.315364 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":148,"threads_started":1}
I20260620 14:26:40.315809 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.308864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36885 (local address 127.0.0.1:37046)
0620 14:26:40.309137 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:40.309177 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:40.309309 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:40.310495 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:40.310500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:40.310520 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:40.310639 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.310646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.312068 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.312071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.313366 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.313376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.314517 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.314521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.314615 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.314634 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:26:40.314697 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:26:40.314768 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":53,"thread_start_us":74,"threads_started":1}
I20260620 14:26:40.318233 10889 tablet_service.cc:1467] Processing CreateTablet for tablet b60bd3659ed149ddbef1eeb37634b8d8 (DEFAULT_TABLE table=test-table [id=a864e70d7e95444bb43d24d750f36e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:40.318233 10755 tablet_service.cc:1467] Processing CreateTablet for tablet b60bd3659ed149ddbef1eeb37634b8d8 (DEFAULT_TABLE table=test-table [id=a864e70d7e95444bb43d24d750f36e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:40.318739 11023 tablet_service.cc:1467] Processing CreateTablet for tablet b60bd3659ed149ddbef1eeb37634b8d8 (DEFAULT_TABLE table=test-table [id=a864e70d7e95444bb43d24d750f36e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:26:40.319288 10755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60bd3659ed149ddbef1eeb37634b8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:40.319286 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60bd3659ed149ddbef1eeb37634b8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:40.319602 11023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60bd3659ed149ddbef1eeb37634b8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:26:40.325630 11106 tablet_bootstrap.cc:492] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: Bootstrap starting.
I20260620 14:26:40.326349 11107 tablet_bootstrap.cc:492] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5: Bootstrap starting.
I20260620 14:26:40.326349 11108 tablet_bootstrap.cc:492] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: Bootstrap starting.
I20260620 14:26:40.327521 11106 tablet_bootstrap.cc:654] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:40.328512 11106 log.cc:824] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:40.328755 11107 tablet_bootstrap.cc:654] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:40.328758 11108 tablet_bootstrap.cc:654] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: Neither blocks nor log segments found. Creating new log.
I20260620 14:26:40.329592 11107 log.cc:824] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:40.329592 11108 log.cc:824] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: Log is configured to *not* fsync() on all Append() calls
I20260620 14:26:40.330338 11106 tablet_bootstrap.cc:492] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: No bootstrap required, opened a new log
I20260620 14:26:40.330530 11106 ts_tablet_manager.cc:1397] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:26:40.331781 11108 tablet_bootstrap.cc:492] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: No bootstrap required, opened a new log
I20260620 14:26:40.331810 11107 tablet_bootstrap.cc:492] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5: No bootstrap required, opened a new log
I20260620 14:26:40.332024 11108 ts_tablet_manager.cc:1397] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:26:40.332482 11107 ts_tablet_manager.cc:1397] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:26:40.334314 11106 raft_consensus.cc:348] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf551fd81694a7899a0f2861fe3690b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36885 } } peers { permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 } } peers { permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 } }
I20260620 14:26:40.334582 11106 raft_consensus.cc:374] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:40.334637 11106 raft_consensus.cc:729] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf551fd81694a7899a0f2861fe3690b, State: Initialized, Role: FOLLOWER
I20260620 14:26:40.335060 11106 consensus_queue.cc:260] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf551fd81694a7899a0f2861fe3690b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36885 } } peers { permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 } } peers { permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 } }
I20260620 14:26:40.335706 11089 heartbeater.cc:503] Master 127.0.98.62:46855 was elected leader, sending a full tablet report...
I20260620 14:26:40.335834 11108 raft_consensus.cc:348] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf551fd81694a7899a0f2861fe3690b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36885 } } peers { permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 } } peers { permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 } }
I20260620 14:26:40.336022 11106 ts_tablet_manager.cc:1428] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:26:40.336061 11108 raft_consensus.cc:374] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:40.336105 11108 raft_consensus.cc:729] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa4bab034064a9ea69914868988a383, State: Initialized, Role: FOLLOWER
I20260620 14:26:40.336141 11107 raft_consensus.cc:348] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf551fd81694a7899a0f2861fe3690b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36885 } } peers { permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 } } peers { permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 } }
I20260620 14:26:40.336359 11107 raft_consensus.cc:374] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:26:40.336403 11107 raft_consensus.cc:729] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12afe5ea4691474fa6b108dd5139d2a5, State: Initialized, Role: FOLLOWER
I20260620 14:26:40.336485 11108 consensus_queue.cc:260] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf551fd81694a7899a0f2861fe3690b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36885 } } peers { permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 } } peers { permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 } }
I20260620 14:26:40.336807 11107 consensus_queue.cc:260] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf551fd81694a7899a0f2861fe3690b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36885 } } peers { permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 } } peers { permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 } }
I20260620 14:26:40.337419 10821 heartbeater.cc:503] Master 127.0.98.62:46855 was elected leader, sending a full tablet report...
I20260620 14:26:40.337670 11107 ts_tablet_manager.cc:1428] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:26:40.338205 11108 ts_tablet_manager.cc:1428] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:26:40.338512 10955 heartbeater.cc:503] Master 127.0.98.62:46855 was elected leader, sending a full tablet report...
W20260620 14:26:40.378456 10822 tablet.cc:2369] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:26:40.477252 11112 raft_consensus.cc:482] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:26:40.477427 11112 raft_consensus.cc:504] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf551fd81694a7899a0f2861fe3690b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36885 } } peers { permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 } } peers { permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 } }
I20260620 14:26:40.478477 11112 leader_election.cc:288] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffa4bab034064a9ea69914868988a383 (127.0.98.2:35663), 12afe5ea4691474fa6b108dd5139d2a5 (127.0.98.1:38793)
I20260620 14:26:40.482513 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.478754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35663 (local address 127.0.98.3:38929)
0620 14:26:40.478884 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:40.478899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:40.478994 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:40.479251 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:40.479254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:40.479274 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:40.479402 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.479407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.480364 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.480367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.481350 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.481359 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.482107 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.482110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.482229 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.482253 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:26:40.482323 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:26:40.482399 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:26:40.482545 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.478844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38929 (local address 127.0.98.2:35663)
0620 14:26:40.478989 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:40.478995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:40.479014 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:26:40.479076 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:40.479082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:40.479136 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:40.479229 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:40.479535 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.480265 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.481463 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.481991 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.482070 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.482117 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:40.482366 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:26:40.482427 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:26:40.482754 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.479052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38793 (local address 127.0.98.3:34877)
0620 14:26:40.479453 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:40.479486 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:26:40.479583 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:26:40.479808 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:26:40.479811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:26:40.479827 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:26:40.479926 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.479932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.480736 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.480739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.481662 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:26:40.481671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.482375 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.482378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:26:40.482470 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.482491 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:26:40.482562 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:26:40.482626 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":156,"threads_started":1}
I20260620 14:26:40.482800 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.479163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34877 (local address 127.0.98.1:38793)
0620 14:26:40.479282 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:40.479286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:40.479580 (+ 294us) server_negotiation.cc:407] Connection header received
0620 14:26:40.479673 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:40.479675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:40.479714 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:40.479772 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:26:40.480026 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.480629 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.481777 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.482244 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.482355 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.482404 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:40.482591 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:26:40.482663 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:26:40.483245 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60bd3659ed149ddbef1eeb37634b8d8" candidate_uuid: "dbf551fd81694a7899a0f2861fe3690b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa4bab034064a9ea69914868988a383" is_pre_election: true
I20260620 14:26:40.483247 10775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60bd3659ed149ddbef1eeb37634b8d8" candidate_uuid: "dbf551fd81694a7899a0f2861fe3690b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12afe5ea4691474fa6b108dd5139d2a5" is_pre_election: true
I20260620 14:26:40.483530 10775 raft_consensus.cc:2454] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf551fd81694a7899a0f2861fe3690b in term 0.
I20260620 14:26:40.483530 10909 raft_consensus.cc:2454] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbf551fd81694a7899a0f2861fe3690b in term 0.
I20260620 14:26:40.483999 10977 leader_election.cc:302] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12afe5ea4691474fa6b108dd5139d2a5, dbf551fd81694a7899a0f2861fe3690b; no voters:
I20260620 14:26:40.484270 11112 raft_consensus.cc:2781] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:26:40.484340 11112 raft_consensus.cc:482] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:26:40.484370 11112 raft_consensus.cc:3037] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:40.485345 11112 raft_consensus.cc:504] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf551fd81694a7899a0f2861fe3690b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36885 } } peers { permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 } } peers { permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 } }
I20260620 14:26:40.485661 11112 leader_election.cc:288] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [CANDIDATE]: Term 1 election: Requested vote from peers ffa4bab034064a9ea69914868988a383 (127.0.98.2:35663), 12afe5ea4691474fa6b108dd5139d2a5 (127.0.98.1:38793)
I20260620 14:26:40.486013 10909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60bd3659ed149ddbef1eeb37634b8d8" candidate_uuid: "dbf551fd81694a7899a0f2861fe3690b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffa4bab034064a9ea69914868988a383"
I20260620 14:26:40.486117 10909 raft_consensus.cc:3037] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:40.486141 10775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60bd3659ed149ddbef1eeb37634b8d8" candidate_uuid: "dbf551fd81694a7899a0f2861fe3690b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12afe5ea4691474fa6b108dd5139d2a5"
I20260620 14:26:40.486270 10775 raft_consensus.cc:3037] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:26:40.487134 10909 raft_consensus.cc:2454] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf551fd81694a7899a0f2861fe3690b in term 1.
I20260620 14:26:40.487285 10775 raft_consensus.cc:2454] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbf551fd81694a7899a0f2861fe3690b in term 1.
I20260620 14:26:40.487425 10977 leader_election.cc:302] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbf551fd81694a7899a0f2861fe3690b, ffa4bab034064a9ea69914868988a383; no voters:
I20260620 14:26:40.487577 11112 raft_consensus.cc:2781] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:26:40.487764 11112 raft_consensus.cc:686] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 1 LEADER]: Becoming Leader. State: Replica: dbf551fd81694a7899a0f2861fe3690b, State: Running, Role: LEADER
I20260620 14:26:40.488075 11112 consensus_queue.cc:237] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf551fd81694a7899a0f2861fe3690b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36885 } } peers { permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 } } peers { permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 } }
I20260620 14:26:40.491060 10628 catalog_manager.cc:5495] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b reported cstate change: term changed from 0 to 1, leader changed from <none> to dbf551fd81694a7899a0f2861fe3690b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dbf551fd81694a7899a0f2861fe3690b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf551fd81694a7899a0f2861fe3690b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 } health_report { overall_health: UNKNOWN } } }
I20260620 14:26:40.501075 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.498083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.0.98.3:36885)
0620 14:26:40.498202 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:26:40.498205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:26:40.498217 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:26:40.498369 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:26:40.498371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:26:40.498422 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:26:40.498500 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:26:40.498698 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.499138 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.499892 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:26:40.500078 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:26:40.500145 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:26:40.500392 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:26:40.500739 (+ 347us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:26:40.500821 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:26:40.500873 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:26:40.500914 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:26:40.506006 10775 raft_consensus.cc:1261] T b60bd3659ed149ddbef1eeb37634b8d8 P 12afe5ea4691474fa6b108dd5139d2a5 [term 1 FOLLOWER]: Refusing update from remote peer dbf551fd81694a7899a0f2861fe3690b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:40.506033 10909 raft_consensus.cc:1261] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383 [term 1 FOLLOWER]: Refusing update from remote peer dbf551fd81694a7899a0f2861fe3690b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:26:40.506608 11112 consensus_queue.cc:1035] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [LEADER]: Connected to new peer: Peer: permanent_uuid: "12afe5ea4691474fa6b108dd5139d2a5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:40.506770 11116 consensus_queue.cc:1035] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffa4bab034064a9ea69914868988a383" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:26:40.514864 11121 mvcc.cc:205] Tried to move back new op lower bound from 7298931099666673664 to 7298931099601911808. Current Snapshot: MvccSnapshot[applied={T|T < 7298931099666673664}]
I20260620 14:26:40.520701 10630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:40.520848 10630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:26:40.523000 10630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 432cf4bb003d4bad8a665fe448dc2314: Sending DeleteTablet for 3 replicas of tablet b60bd3659ed149ddbef1eeb37634b8d8
I20260620 14:26:40.523654 11023 tablet_service.cc:1514] Processing DeleteTablet for tablet b60bd3659ed149ddbef1eeb37634b8d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37046
I20260620 14:26:40.523731 10755 tablet_service.cc:1514] Processing DeleteTablet for tablet b60bd3659ed149ddbef1eeb37634b8d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38892
I20260620 14:26:40.523756 10889 tablet_service.cc:1514] Processing DeleteTablet for tablet b60bd3659ed149ddbef1eeb37634b8d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:26:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52572
I20260620 14:26:40.524140 11130 tablet_replica.cc:330] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: stopping tablet replica
I20260620 14:26:40.524310 11128 tablet_replica.cc:330] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: stopping tablet replica
I20260620 14:26:40.524408 11130 raft_consensus.cc:2229] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:26:40.524645 11128 raft_consensus.cc:2229] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:26:40.524688 11130 raft_consensus.cc:2258] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:40.524750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 10692
I20260620 14:26:40.525097 11128 raft_consensus.cc:2258] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:26:40.526080 11130 ts_tablet_manager.cc:1905] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:40.526708 11128 ts_tablet_manager.cc:1905] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:26:40.528695 11130 ts_tablet_manager.cc:1918] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:40.528791 11130 log.cc:1196] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/b60bd3659ed149ddbef1eeb37634b8d8
I20260620 14:26:40.529140 11130 ts_tablet_manager.cc:1939] T b60bd3659ed149ddbef1eeb37634b8d8 P ffa4bab034064a9ea69914868988a383: Deleting consensus metadata
I20260620 14:26:40.529618 11128 ts_tablet_manager.cc:1918] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:26:40.529714 11128 log.cc:1196] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/b60bd3659ed149ddbef1eeb37634b8d8
I20260620 14:26:40.530002 11128 ts_tablet_manager.cc:1939] T b60bd3659ed149ddbef1eeb37634b8d8 P dbf551fd81694a7899a0f2861fe3690b: Deleting consensus metadata
I20260620 14:26:40.530289 10615 catalog_manager.cc:4841] TS ffa4bab034064a9ea69914868988a383 (127.0.98.2:35663): tablet b60bd3659ed149ddbef1eeb37634b8d8 (table test-table [id=a864e70d7e95444bb43d24d750f36e1d]) successfully deleted
I20260620 14:26:40.530961 10616 catalog_manager.cc:4841] TS dbf551fd81694a7899a0f2861fe3690b (127.0.98.3:36885): tablet b60bd3659ed149ddbef1eeb37634b8d8 (table test-table [id=a864e70d7e95444bb43d24d750f36e1d]) successfully deleted
W20260620 14:26:40.535408 10615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38793 (error 108)
I20260620 14:26:40.535534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 10826
I20260620 14:26:40.537657 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:26:40.537347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38793 (local address 127.0.0.1:38902)
0620 14:26:40.537490 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:26:40.537565 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:26:40.537823 10615 catalog_manager.cc:4584] TS 12afe5ea4691474fa6b108dd5139d2a5 (127.0.98.1:38793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b60bd3659ed149ddbef1eeb37634b8d8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38793: connect: Connection refused (error 111)
I20260620 14:26:40.541566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 10960
I20260620 14:26:40.546618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 10598
2026-06-20T14:26:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2197 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:26:40 dist-test-slave-pk3j krb5kdc[11134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:26:40 dist-test-slave-pk3j krb5kdc[11134](info): set up 2 sockets
Jun 20 14:26:40 dist-test-slave-pk3j krb5kdc[11134](info): commencing operation
krb5kdc: starting...
W20260620 14:26:41.565877 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:26:41 dist-test-slave-pk3j krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:26:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:26:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:26:41.711958 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDwUWGk/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:26:43.852 UTC [11164] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:26:43.852 UTC [11164] LOG: listening on IPv4 address "127.0.98.20", port 54731
2026-06-20 14:26:43.855 UTC [11164] LOG: listening on Unix socket "/tmp/.s.PGSQL.54731"
2026-06-20 14:26:43.875 UTC [11167] LOG: database system was shut down at 2026-06-20 14:26:42 UTC
2026-06-20 14:26:43.878 UTC [11164] LOG: database system is ready to accept connections
I20260620 14:26:44.829840 392 mini_postgres.cc:96] Postgres bound to 54731
2026-06-20 14:26:44.835 UTC [11175] FATAL: database "slave" does not exist
127.0.98.20:54731 - accepting connections
I20260620 14:26:44.836282 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:26:44.849785 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:26:45.234023 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:26:45.234119 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-admin
2026-06-20 14:26:45,497 [I] DB FLAVOR :POSTGRES
2026-06-20 14:26:45,498 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:26:45,498 [I] Checking connection..
2026-06-20 14:26:45,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:26:45,718 [I] Checking connection passed.
2026-06-20 14:26:45,718 [I] --------- Verifying version history table ---------
2026-06-20 14:26:45,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:26:45,968 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:26:45,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:26:46,187 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:26:46,187 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:26:46,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:26:46.419 UTC [11239] WARNING: there is no transaction in progress
2026-06-20 14:26:46,425 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:26:46,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:26:46,639 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:26:46,639 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:26:46,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:26:46,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:26:47,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pk3j.c.gcp-upstream.internal','N') ;"
2026-06-20 14:26:47,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:26:47,543 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:26:47,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:26:47,755 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:26:47,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:26:48,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:26:48,242 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:26:48,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:26:49.104 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.119 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.133 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.146 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.160 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.178 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.232 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.238 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.251 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.260 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.269 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.282 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.290 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.296 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.302 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49.308 UTC [11347] WARNING: there is no transaction in progress
2026-06-20 14:26:49,853 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:26:49,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:26:50,095 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:26:50,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:26:50,343 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:26:50,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:26:50,559 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:26:50,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:26:50,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:26:51,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pk3j.c.gcp-upstream.internal';"
2026-06-20 14:26:51,231 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:26:51,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:26:51,452 [I] DB_PATCHES have already been applied
I20260620 14:26:51.457624 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDwUWGk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:26:51.457700 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:26:51.460960 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:47575
Jun 20, 2026 2:26:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:26:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:26:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:26:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:26:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:26:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:26:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:26:52.529124 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.693s user 0.000s sys 0.009s
Jun 20, 2026 2:27:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:27:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:27:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:27:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:27:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:27:11.840205 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:27:11 dist-test-slave-pk3j krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965631, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:27:11.914358 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:27:11.922524 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:27:12.724217 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:27:12.724292 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-kms
I20260620 14:27:12.724385 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54731
2026-06-20 14:27:12,902 [I] DB FLAVOR :POSTGRES
2026-06-20 14:27:12,902 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:27:12,903 [I] Checking connection
2026-06-20 14:27:12,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:27:13,127 [I] connection success
2026-06-20 14:27:13,127 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:27:13,127 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:27:13,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:27:13,345 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:27:13,346 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:27:13,346 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:27:13,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54731/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:27:13,588 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:27:14.009166 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:27:14.183670 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:27:14.224710 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:27:14 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965631, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:27:14.623782 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:27:14.624101 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:27:14.624130 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:27:14.627558 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11527
I20260620 14:27:14.627691 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:42705
Jun 20, 2026 2:27:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:27:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:27:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:27:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:27:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:27:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 14:27:16 dist-test-slave-pk3j krb5kdc[11134](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965636, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:27:16.833899 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.920s user 0.006s sys 0.158s
I20260620 14:27:16.834026 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:27:16.834134 392 mini_ranger_kms.cc:401] 127.0.98.20:42705/kms/v1/keys
Jun 20 14:27:17 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965636, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:27:17 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965636, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:27:19 dist-test-slave-pk3j krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:27:19.626145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34241
--ranger_config_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42705/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:27:19.731473 11578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:27:19.731714 11578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:27:19.731750 11578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:27:19.735776 11578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:27:19.735823 11578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:27:19.735841 11578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:27:19.735862 11578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:27:19.735921 11578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:27:19.741074 11578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:42705/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34241
--ranger_config_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:27:19.741989 11578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:27:19.742918 11578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:27:19.751852 11583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:27:19.751956 11584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:27:19.751827 11586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:27:19.752259 11578 server_base.cc:1034] running on GCE node
I20260620 14:27:19.752764 11578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:27:19.753831 11578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:27:19.755015 11578 hybrid_clock.cc:648] HybridClock initialized: now 1781965639754982 us; error 44 us; skew 500 ppm
Jun 20 14:27:19 dist-test-slave-pk3j krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:27:19.762051 11578 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:27:19.762957 11578 webserver.cc:460] Webserver started at http://127.0.98.62:41683/ using document root <none> and password file <none>
I20260620 14:27:19.763463 11578 fs_manager.cc:359] Metadata directory not provided
I20260620 14:27:19.763514 11578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:27:19.763720 11578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:27:19 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:27:19.808954 11578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "32ae177b40d1420faa44157df247367a"
format_stamp: "Formatted at 2026-06-20 14:27:19 on dist-test-slave-pk3j"
server_key: "67a9555a3d91a10b837b1409280fbee6"
server_key_iv: "5aa1e9847b7a3dc70497ab1f2f774c4d"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:19.809695 11578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "32ae177b40d1420faa44157df247367a"
format_stamp: "Formatted at 2026-06-20 14:27:19 on dist-test-slave-pk3j"
server_key: "67a9555a3d91a10b837b1409280fbee6"
server_key_iv: "5aa1e9847b7a3dc70497ab1f2f774c4d"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:19.814884 11578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 14:27:19.830588 11593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:27:19.831971 11578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:27:19.832119 11578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "32ae177b40d1420faa44157df247367a"
format_stamp: "Formatted at 2026-06-20 14:27:19 on dist-test-slave-pk3j"
server_key: "67a9555a3d91a10b837b1409280fbee6"
server_key_iv: "5aa1e9847b7a3dc70497ab1f2f774c4d"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:19.832254 11578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:27:19.847204 11578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:27:19.849869 11578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:27:19.850059 11578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:27:19.857581 11578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34241
I20260620 14:27:19.857604 11645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34241 every 8 connection(s)
I20260620 14:27:19.858487 11578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:27:19.861545 11646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:27:19.862082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 11578
I20260620 14:27:19.862187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:27:19.862485 392 external_mini_cluster.cc:1428] Setting key 4d837f7017bb8b21a9513e23022594cc
I20260620 14:27:19.867914 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a: Bootstrap starting.
I20260620 14:27:19.870235 11646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a: Neither blocks nor log segments found. Creating new log.
I20260620 14:27:19.871127 11646 log.cc:824] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:27:19 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:27:19.872891 11646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a: No bootstrap required, opened a new log
I20260620 14:27:19.876926 11646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ae177b40d1420faa44157df247367a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260620 14:27:19.877257 11646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:27:19.877394 11646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ae177b40d1420faa44157df247367a, State: Initialized, Role: FOLLOWER
I20260620 14:27:19.877924 11646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ae177b40d1420faa44157df247367a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260620 14:27:19.878085 11646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:27:19.878167 11646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:27:19.878255 11646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:27:19.878605 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:19.863874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.0.98.62:34241)
0620 14:27:19.864311 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:19.864329 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:19.864371 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:27:19.865164 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:19.865185 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:19.865596 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:19.865933 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:27:19.866727 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:19.868164 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:19.869398 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:19.870011 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:19.870178 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:19.873018 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:27:19.873025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:27:19.873041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:27:19.873068 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:27:19.875638 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:27:19.876118 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:27:19.876124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:27:19.876130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:27:19.876253 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:27:19.876547 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:27:19.876550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:27:19.876552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:27:19.876995 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:27:19.877185 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:19.877475 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:27:19.877724 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":130,"threads_started":1}
I20260620 14:27:19.879159 11646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ae177b40d1420faa44157df247367a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260620 14:27:19.879491 11646 leader_election.cc:302] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32ae177b40d1420faa44157df247367a; no voters:
I20260620 14:27:19.879799 11646 leader_election.cc:288] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:27:19.879866 11651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:27:19.880112 11651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [term 1 LEADER]: Becoming Leader. State: Replica: 32ae177b40d1420faa44157df247367a, State: Running, Role: LEADER
I20260620 14:27:19.880465 11651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ae177b40d1420faa44157df247367a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260620 14:27:19.881013 11646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:27:19.881611 11651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ae177b40d1420faa44157df247367a. Latest consensus state: current_term: 1 leader_uuid: "32ae177b40d1420faa44157df247367a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ae177b40d1420faa44157df247367a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } } }
I20260620 14:27:19.881822 11651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [sys.catalog]: This master's current role is: LEADER
I20260620 14:27:19.882076 11651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32ae177b40d1420faa44157df247367a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ae177b40d1420faa44157df247367a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } } }
I20260620 14:27:19.882216 11651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a [sys.catalog]: This master's current role is: LEADER
I20260620 14:27:19.882491 11659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:27:19.885478 11659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:27:19.885886 11646 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:27:19.897730 11659 catalog_manager.cc:1347] Generated new cluster ID: ebd2106f81d74e5fb291ffd58f0c92b1
I20260620 14:27:19.897797 11659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:27:19.910233 11659 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:27:19.911494 11659 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:27:19.916906 11659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a: Generated new TSK 0
I20260620 14:27:19.917694 11659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:27:20.624580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:27:20.779876 11691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:27:20.780186 11691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:27:20.780246 11691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:27:20.786249 11691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:27:20.786329 11691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:27:20.786413 11691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:27:20.794584 11691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:42705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:27:20.795822 11691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:27:20.797047 11691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:27:20.808835 11697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:27:20.808890 11696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:27:20.808831 11699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:27:20.809311 11691 server_base.cc:1034] running on GCE node
I20260620 14:27:20.809677 11691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:27:20.810365 11691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:27:20.813704 11691 hybrid_clock.cc:648] HybridClock initialized: now 1781965640813682 us; error 31 us; skew 500 ppm
Jun 20 14:27:20 dist-test-slave-pk3j krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:27:20.816928 11691 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:27:20.817839 11691 webserver.cc:460] Webserver started at http://127.0.98.1:43923/ using document root <none> and password file <none>
I20260620 14:27:20.818495 11691 fs_manager.cc:359] Metadata directory not provided
I20260620 14:27:20.818624 11691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:27:20.818868 11691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:27:20 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:27:20.837003 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "51926d98851c40f38b2828e2add3fc5e"
format_stamp: "Formatted at 2026-06-20 14:27:20 on dist-test-slave-pk3j"
server_key: "f2e71a8e4182eb289fea45d6e32602d2"
server_key_iv: "c1ca60e42cab248168aff93233fc6499"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:20.838191 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "51926d98851c40f38b2828e2add3fc5e"
format_stamp: "Formatted at 2026-06-20 14:27:20 on dist-test-slave-pk3j"
server_key: "f2e71a8e4182eb289fea45d6e32602d2"
server_key_iv: "c1ca60e42cab248168aff93233fc6499"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:20.843215 11691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:27:20.867163 11708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:27:20.868628 11691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:27:20.868728 11691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "51926d98851c40f38b2828e2add3fc5e"
format_stamp: "Formatted at 2026-06-20 14:27:20 on dist-test-slave-pk3j"
server_key: "f2e71a8e4182eb289fea45d6e32602d2"
server_key_iv: "c1ca60e42cab248168aff93233fc6499"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:20.868834 11691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:27:20.887009 11691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:27:20.890677 11691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:27:20.890944 11691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:27:20.891670 11691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:27:20.892704 11691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:27:20.892805 11691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:27:20.892936 11691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:27:20.893028 11691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:27:20.903137 11691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40931
I20260620 14:27:20.904222 11691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:27:20.912987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 11691
I20260620 14:27:20.913110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:27:20.913429 392 external_mini_cluster.cc:1428] Setting key d8cd30a46ba8c102b5c06ffcc90c28f8
I20260620 14:27:20.914429 11821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40931 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 20 14:27:20 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:27:20.938401 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:20.916814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:40791)
0620 14:27:20.919287 (+ 2473us) negotiation.cc:107] Waiting for socket to connect
0620 14:27:20.919338 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:27:20.920346 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:27:20.921096 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:27:20.921104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:27:20.921466 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:27:20.922092 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:20.922111 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:20.923302 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:20.923305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:20.924629 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:20.924639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:20.925407 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:20.925411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:20.925594 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:20.929380 (+ 3786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:27:20.929427 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:27:20.932898 (+ 3471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:27:20.935285 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:27:20.935293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:27:20.935308 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:27:20.935634 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:27:20.935948 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:27:20.935951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:27:20.935953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:27:20.936112 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:27:20.936624 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:27:20.936630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:27:20.936925 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 14:27:20.937119 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 14:27:20.937453 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2252,"thread_start_us":119,"threads_started":1}
I20260620 14:27:20.939159 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:20.916320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40791 (local address 127.0.98.62:34241)
0620 14:27:20.917805 (+ 1485us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:20.917815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:20.919429 (+ 1614us) server_negotiation.cc:407] Connection header received
0620 14:27:20.920579 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:20.920582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:20.920640 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:20.920702 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:27:20.922261 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:20.923139 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:20.924767 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:20.925098 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:20.925178 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:20.933114 (+ 7936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:27:20.933118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:27:20.933121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:27:20.933147 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:27:20.935163 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:27:20.935753 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:27:20.935757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:27:20.935760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:27:20.935846 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:27:20.936237 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:27:20.936240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:27:20.936242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:27:20.936470 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:27:20.936559 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:20.938877 (+ 2318us) server_negotiation.cc:299] Negotiation successful
0620 14:27:20.938999 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1409,"thread_start_us":106,"threads_started":1}
I20260620 14:27:20.940739 11822 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260620 14:27:20.940956 11822 heartbeater.cc:464] Registering TS with master...
I20260620 14:27:20.941520 11822 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260620 14:27:20.943181 11610 ts_manager.cc:194] Registered new tserver with Master: 51926d98851c40f38b2828e2add3fc5e (127.0.98.1:40931)
I20260620 14:27:20.944617 11610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:27:20.960048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:27:21 dist-test-slave-pk3j krb5kdc[11134](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:27:21.077838 11830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:27:21.078091 11830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:27:21.078189 11830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:27:21.083029 11830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:27:21.083151 11830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:27:21.083271 11830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:27:21.089051 11830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:42705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:27:21.090209 11830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:27:21.091197 11830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:27:21.101368 11839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:27:21.101593 11836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:27:21.101368 11837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:27:21.102789 11830 server_base.cc:1034] running on GCE node
I20260620 14:27:21.103168 11830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:27:21.103848 11830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:27:21.105067 11830 hybrid_clock.cc:648] HybridClock initialized: now 1781965641105034 us; error 45 us; skew 500 ppm
Jun 20 14:27:21 dist-test-slave-pk3j krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:27:21.107872 11830 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:27:21.108737 11830 webserver.cc:460] Webserver started at http://127.0.98.2:43571/ using document root <none> and password file <none>
I20260620 14:27:21.109346 11830 fs_manager.cc:359] Metadata directory not provided
I20260620 14:27:21.109439 11830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:27:21.109635 11830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:27:21 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:27:21.127992 11830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "d35e9604e21b49c690e8497247e11ce1"
format_stamp: "Formatted at 2026-06-20 14:27:21 on dist-test-slave-pk3j"
server_key: "fadcdcc50865dd558bba5fc9c12cf677"
server_key_iv: "cd77f72241ecfb456c5be6088a30a917"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:21.128647 11830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d35e9604e21b49c690e8497247e11ce1"
format_stamp: "Formatted at 2026-06-20 14:27:21 on dist-test-slave-pk3j"
server_key: "fadcdcc50865dd558bba5fc9c12cf677"
server_key_iv: "cd77f72241ecfb456c5be6088a30a917"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:21.132513 11830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:27:21.153609 11846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:27:21.155215 11830 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260620 14:27:21.155372 11830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "d35e9604e21b49c690e8497247e11ce1"
format_stamp: "Formatted at 2026-06-20 14:27:21 on dist-test-slave-pk3j"
server_key: "fadcdcc50865dd558bba5fc9c12cf677"
server_key_iv: "cd77f72241ecfb456c5be6088a30a917"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:21.155530 11830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:27:21.170590 11830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:27:21.174551 11830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:27:21.174726 11830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:27:21.175355 11830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:27:21.176301 11830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:27:21.176373 11830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:27:21.176435 11830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:27:21.176489 11830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:27:21.187793 11830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37017
I20260620 14:27:21.188917 11830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:27:21.189118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 11830
I20260620 14:27:21.189253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:27:21.189596 392 external_mini_cluster.cc:1428] Setting key d0f6f6ef224ff77fa19075e3eb06dc5d
I20260620 14:27:21.197844 11959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37017 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Jun 20 14:27:21 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:27:21.229241 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.202989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45085 (local address 127.0.98.62:34241)
0620 14:27:21.203125 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:21.203129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:21.213766 (+ 10637us) server_negotiation.cc:407] Connection header received
0620 14:27:21.215062 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:21.215066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:21.215123 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:21.215175 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:27:21.216798 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.217618 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.219327 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.219674 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.219748 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.224881 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:27:21.224885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:27:21.224888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:27:21.224917 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:27:21.226762 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:27:21.227380 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:27:21.227384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:27:21.227385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:27:21.227469 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:27:21.227896 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:27:21.227900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:27:21.227902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:27:21.228117 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:27:21.228187 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:21.228889 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 14:27:21.229017 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:27:21.230819 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.203521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:45085)
0620 14:27:21.213579 (+ 10058us) negotiation.cc:107] Waiting for socket to connect
0620 14:27:21.213643 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 14:27:21.214797 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:27:21.215594 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:27:21.215603 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:27:21.215973 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:27:21.216622 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.216639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.217769 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.217772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.219192 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.219202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.219948 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.219952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.220134 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.220828 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:27:21.220848 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:27:21.224676 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:27:21.226913 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:27:21.226920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:27:21.226936 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:27:21.227260 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:27:21.227578 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:27:21.227581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:27:21.227583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:27:21.227768 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:27:21.228267 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:27:21.228273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:27:21.228591 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 14:27:21.229713 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0620 14:27:21.229973 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9804,"thread_start_us":142,"threads_started":1}
I20260620 14:27:21.232004 11960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260620 14:27:21.232244 11960 heartbeater.cc:464] Registering TS with master...
I20260620 14:27:21.232748 11960 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260620 14:27:21.233798 11610 ts_manager.cc:194] Registered new tserver with Master: d35e9604e21b49c690e8497247e11ce1 (127.0.98.2:37017)
I20260620 14:27:21.234366 11610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:27:21.249694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42705/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:27:21.411139 11965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:27:21.411497 11965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:27:21.411584 11965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:27:21.417994 11965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:27:21.418071 11965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:27:21.418172 11965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:27:21.425760 11965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:42705/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:27:21.427037 11965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:27:21.428436 11965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:27:21.441684 11971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:27:21.442046 11970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:27:21.442082 11973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:27:21.442422 11965 server_base.cc:1034] running on GCE node
I20260620 14:27:21.443037 11965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:27:21.443799 11965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:27:21.445262 11965 hybrid_clock.cc:648] HybridClock initialized: now 1781965641445167 us; error 86 us; skew 500 ppm
Jun 20 14:27:21 dist-test-slave-pk3j krb5kdc[11134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:27:21.448968 11965 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:27:21.450212 11965 webserver.cc:460] Webserver started at http://127.0.98.3:39321/ using document root <none> and password file <none>
I20260620 14:27:21.451063 11965 fs_manager.cc:359] Metadata directory not provided
I20260620 14:27:21.451143 11965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:27:21.451375 11965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:27:21 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:27:21.485334 11965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "576d5f8e17b1424a869b1f5400c39208"
format_stamp: "Formatted at 2026-06-20 14:27:21 on dist-test-slave-pk3j"
server_key: "06a8ecc5366b38f737650ca95579e8dc"
server_key_iv: "70038d6d33d0e443ed711eed64b426dc"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:21.486131 11965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "576d5f8e17b1424a869b1f5400c39208"
format_stamp: "Formatted at 2026-06-20 14:27:21 on dist-test-slave-pk3j"
server_key: "06a8ecc5366b38f737650ca95579e8dc"
server_key_iv: "70038d6d33d0e443ed711eed64b426dc"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:21.502105 11965 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.005s sys 0.000s
Jun 20 14:27:21 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:27:21.523866 11980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:27:21.525470 11965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:27:21.525596 11965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "576d5f8e17b1424a869b1f5400c39208"
format_stamp: "Formatted at 2026-06-20 14:27:21 on dist-test-slave-pk3j"
server_key: "06a8ecc5366b38f737650ca95579e8dc"
server_key_iv: "70038d6d33d0e443ed711eed64b426dc"
server_key_version: "kuduclusterkey@0"
I20260620 14:27:21.525727 11965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:27:21.563619 11965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:27:21.567189 11965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:27:21.567369 11965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:27:21.568029 11965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:27:21.568979 11965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:27:21.569036 11965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:27:21.569088 11965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:27:21.569118 11965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:27:21.582460 11965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35555
I20260620 14:27:21.583657 11965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:27:21.585383 12093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35555 every 8 connection(s)
I20260620 14:27:21.590108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 11965
I20260620 14:27:21.590229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:27:21.590523 392 external_mini_cluster.cc:1428] Setting key 2c82c6ef1c4112dd1d4f26837f53c2f6
Jun 20 14:27:21 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:27:21.608031 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.587319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34345 (local address 127.0.98.62:34241)
0620 14:27:21.587457 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:21.587461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:21.589797 (+ 2336us) server_negotiation.cc:407] Connection header received
0620 14:27:21.592763 (+ 2966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:21.592767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:21.592820 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:21.592870 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:27:21.594507 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.595346 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.596978 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.597363 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.597433 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.603378 (+ 5945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:27:21.603382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:27:21.603385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:27:21.603412 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:27:21.605655 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:27:21.606235 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:27:21.606239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:27:21.606241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:27:21.606324 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:27:21.606685 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:27:21.606688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:27:21.606690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:27:21.606899 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:27:21.606976 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:21.607616 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 14:27:21.607730 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:27:21.609583 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.587800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:34345)
0620 14:27:21.589612 (+ 1812us) negotiation.cc:107] Waiting for socket to connect
0620 14:27:21.589666 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:27:21.591550 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:27:21.593320 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:27:21.593330 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:27:21.593680 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:27:21.594331 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.594349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.595499 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.595502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.596847 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.596857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.597618 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.597621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.597807 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.601380 (+ 3573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:27:21.601400 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:27:21.603196 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:27:21.605780 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:27:21.605787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:27:21.605802 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:27:21.606123 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:27:21.606426 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:27:21.606429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:27:21.606431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:27:21.606587 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:27:21.607040 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:27:21.607046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:27:21.607347 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 14:27:21.608246 (+ 899us) client_negotiation.cc:240] Negotiation successful
0620 14:27:21.608566 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1581,"thread_start_us":150,"threads_started":1}
I20260620 14:27:21.610731 12094 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260620 14:27:21.610984 12094 heartbeater.cc:464] Registering TS with master...
I20260620 14:27:21.611501 12094 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260620 14:27:21.612545 11610 ts_manager.cc:194] Registered new tserver with Master: 576d5f8e17b1424a869b1f5400c39208 (127.0.98.3:35555)
I20260620 14:27:21.613269 11610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34345
I20260620 14:27:21.616686 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:27:21.628427 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.619045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.0.98.62:34241)
0620 14:27:21.619159 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:21.619163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:21.619540 (+ 377us) server_negotiation.cc:407] Connection header received
0620 14:27:21.619703 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:21.619706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:21.619746 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:21.619792 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:27:21.620688 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.621513 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.622831 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.623166 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.623238 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.624954 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:27:21.624958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:27:21.624961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:27:21.624985 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:27:21.626688 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:27:21.627159 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:27:21.627162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:27:21.627164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:27:21.627240 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:27:21.627532 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:27:21.627535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:27:21.627536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:27:21.627722 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:27:21.627819 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:21.628154 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 14:27:21.628262 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:27:21.632031 11610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:27:21.636181 11610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:27:21.658053 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.648508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37017 (local address 127.0.0.1:43240)
0620 14:27:21.650364 (+ 1856us) negotiation.cc:107] Waiting for socket to connect
0620 14:27:21.650374 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:27:21.650433 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:27:21.651957 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:27:21.651963 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:27:21.651984 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:27:21.652092 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.652098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.653631 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.653634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.655663 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.655674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.657503 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.657507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.657612 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.657650 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:27:21.657749 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:27:21.657898 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1799,"thread_start_us":80,"threads_started":1}
I20260620 14:27:21.659003 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.648848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.0.98.2:37017)
0620 14:27:21.650636 (+ 1788us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:21.650650 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:21.650672 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:27:21.650742 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:21.650747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:21.650963 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:21.651794 (+ 831us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:27:21.652217 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.653504 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.655845 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.656610 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.656714 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.658702 (+ 1988us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:21.658753 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:27:21.658868 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1687,"thread_start_us":77,"threads_started":1}
I20260620 14:27:21.662465 11878 tablet_service.cc:1467] Processing CreateTablet for tablet e2809e174284487486908f4159d53768 (DEFAULT_TABLE table=test-table [id=25865e215b1c40519448456e9f8f75da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:27:21.663606 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2809e174284487486908f4159d53768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:27:21.666548 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.646510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40931 (local address 127.0.0.1:38652)
0620 14:27:21.649793 (+ 3283us) negotiation.cc:107] Waiting for socket to connect
0620 14:27:21.649838 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:27:21.649952 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:27:21.657879 (+ 7927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:27:21.657882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:27:21.657898 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:27:21.657975 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.657981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.661739 (+ 3758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.661743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.662879 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.662890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.666185 (+ 3295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.666189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.666275 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.666298 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:27:21.666349 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:27:21.666412 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3122,"thread_start_us":88,"threads_started":1}
I20260620 14:27:21.667445 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.647355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.0.98.1:40931)
0620 14:27:21.657409 (+ 10054us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:21.657417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:21.657443 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:27:21.657517 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:21.657522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:21.657730 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:21.658125 (+ 395us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:27:21.658286 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.659373 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.663039 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.663808 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.667021 (+ 3213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.667171 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:21.667222 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:27:21.667315 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9939,"thread_start_us":98,"threads_started":1}
I20260620 14:27:21.668818 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.647914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35555 (local address 127.0.0.1:39466)
0620 14:27:21.650144 (+ 2230us) negotiation.cc:107] Waiting for socket to connect
0620 14:27:21.650155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:27:21.650215 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:27:21.660060 (+ 9845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:27:21.660063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:27:21.660082 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:27:21.660163 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.660168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.663956 (+ 3788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.663959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.665039 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.665053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.668476 (+ 3423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.668480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.668565 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.668587 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:27:21.668637 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:27:21.668697 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2179,"thread_start_us":78,"threads_started":1}
I20260620 14:27:21.669689 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.648057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.0.98.3:35555)
0620 14:27:21.659633 (+ 11576us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:21.659644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:21.659665 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:27:21.659729 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:21.659734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:21.659905 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:21.660316 (+ 411us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:27:21.660442 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.661578 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.665257 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.666042 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.669276 (+ 3234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.669412 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:21.669462 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:27:21.669550 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11477,"thread_start_us":106,"threads_started":1}
I20260620 14:27:21.670441 11756 tablet_service.cc:1467] Processing CreateTablet for tablet e2809e174284487486908f4159d53768 (DEFAULT_TABLE table=test-table [id=25865e215b1c40519448456e9f8f75da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:27:21.671507 11756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2809e174284487486908f4159d53768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:27:21.673635 12009 tablet_service.cc:1467] Processing CreateTablet for tablet e2809e174284487486908f4159d53768 (DEFAULT_TABLE table=test-table [id=25865e215b1c40519448456e9f8f75da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:27:21.674196 12112 tablet_bootstrap.cc:492] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: Bootstrap starting.
I20260620 14:27:21.674549 12009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2809e174284487486908f4159d53768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:27:21.677130 12112 tablet_bootstrap.cc:654] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: Neither blocks nor log segments found. Creating new log.
I20260620 14:27:21.678841 12112 log.cc:824] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:27:21.680805 12112 tablet_bootstrap.cc:492] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: No bootstrap required, opened a new log
I20260620 14:27:21.680995 12112 ts_tablet_manager.cc:1397] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:27:21.683682 12115 tablet_bootstrap.cc:492] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: Bootstrap starting.
I20260620 14:27:21.686175 12115 tablet_bootstrap.cc:654] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: Neither blocks nor log segments found. Creating new log.
I20260620 14:27:21.686931 12115 log.cc:824] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: Log is configured to *not* fsync() on all Append() calls
I20260620 14:27:21.687553 12113 tablet_bootstrap.cc:492] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e: Bootstrap starting.
I20260620 14:27:21.689553 12113 tablet_bootstrap.cc:654] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e: Neither blocks nor log segments found. Creating new log.
I20260620 14:27:21.690287 12113 log.cc:824] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:27:21.693681 12112 raft_consensus.cc:348] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 } } peers { permanent_uuid: "d35e9604e21b49c690e8497247e11ce1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37017 } }
I20260620 14:27:21.693941 12112 raft_consensus.cc:374] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:27:21.693990 12112 raft_consensus.cc:729] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35e9604e21b49c690e8497247e11ce1, State: Initialized, Role: FOLLOWER
I20260620 14:27:21.694432 12112 consensus_queue.cc:260] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 } } peers { permanent_uuid: "d35e9604e21b49c690e8497247e11ce1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37017 } }
I20260620 14:27:21.695412 12112 ts_tablet_manager.cc:1428] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260620 14:27:21.695835 11960 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
I20260620 14:27:21.696432 12115 tablet_bootstrap.cc:492] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: No bootstrap required, opened a new log
I20260620 14:27:21.697397 12115 ts_tablet_manager.cc:1397] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260620 14:27:21.701823 12115 raft_consensus.cc:348] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 } } peers { permanent_uuid: "d35e9604e21b49c690e8497247e11ce1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37017 } }
I20260620 14:27:21.702144 12115 raft_consensus.cc:374] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:27:21.702243 12115 raft_consensus.cc:729] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576d5f8e17b1424a869b1f5400c39208, State: Initialized, Role: FOLLOWER
I20260620 14:27:21.702796 12115 consensus_queue.cc:260] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 } } peers { permanent_uuid: "d35e9604e21b49c690e8497247e11ce1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37017 } }
I20260620 14:27:21.703871 12115 ts_tablet_manager.cc:1428] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:27:21.704249 12094 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
W20260620 14:27:21.705222 11961 tablet.cc:2369] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:27:21.711011 12113 tablet_bootstrap.cc:492] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e: No bootstrap required, opened a new log
I20260620 14:27:21.711216 12113 ts_tablet_manager.cc:1397] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20260620 14:27:21.715878 12113 raft_consensus.cc:348] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 } } peers { permanent_uuid: "d35e9604e21b49c690e8497247e11ce1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37017 } }
I20260620 14:27:21.716126 12113 raft_consensus.cc:374] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:27:21.716173 12113 raft_consensus.cc:729] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51926d98851c40f38b2828e2add3fc5e, State: Initialized, Role: FOLLOWER
I20260620 14:27:21.716600 12113 consensus_queue.cc:260] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 } } peers { permanent_uuid: "d35e9604e21b49c690e8497247e11ce1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37017 } }
I20260620 14:27:21.717574 12113 ts_tablet_manager.cc:1428] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:27:21.717891 11822 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
I20260620 14:27:21.791004 12118 raft_consensus.cc:482] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:27:21.791244 12118 raft_consensus.cc:504] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 } } peers { permanent_uuid: "d35e9604e21b49c690e8497247e11ce1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37017 } }
I20260620 14:27:21.792552 12118 leader_election.cc:288] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51926d98851c40f38b2828e2add3fc5e (127.0.98.1:40931), 576d5f8e17b1424a869b1f5400c39208 (127.0.98.3:35555)
I20260620 14:27:21.798144 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.793286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48901 (local address 127.0.98.3:35555)
0620 14:27:21.793435 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:21.793440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:21.793851 (+ 411us) server_negotiation.cc:407] Connection header received
0620 14:27:21.794030 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:21.794033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:21.794074 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:21.794115 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:27:21.794472 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.795221 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.796814 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.797456 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.797548 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.797612 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:21.797939 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 14:27:21.798016 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:27:21.798659 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.792904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35555 (local address 127.0.98.2:48901)
0620 14:27:21.793775 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0620 14:27:21.793788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:27:21.793922 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:27:21.794234 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:27:21.794237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:27:21.794250 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:27:21.794353 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.794359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.795388 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.795391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.796670 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.796681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.797703 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.797707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.797801 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.797824 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:27:21.798331 (+ 507us) client_negotiation.cc:240] Negotiation successful
0620 14:27:21.798409 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":79,"threads_started":1}
I20260620 14:27:21.799324 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2809e174284487486908f4159d53768" candidate_uuid: "d35e9604e21b49c690e8497247e11ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576d5f8e17b1424a869b1f5400c39208" is_pre_election: true
I20260620 14:27:21.799659 12047 raft_consensus.cc:2454] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35e9604e21b49c690e8497247e11ce1 in term 0.
I20260620 14:27:21.800138 11847 leader_election.cc:302] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576d5f8e17b1424a869b1f5400c39208, d35e9604e21b49c690e8497247e11ce1; no voters:
I20260620 14:27:21.800957 12118 raft_consensus.cc:2781] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:27:21.801054 12118 raft_consensus.cc:482] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:27:21.801124 12118 raft_consensus.cc:3037] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:27:21.802474 12118 raft_consensus.cc:504] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 } } peers { permanent_uuid: "d35e9604e21b49c690e8497247e11ce1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37017 } }
I20260620 14:27:21.802810 12118 leader_election.cc:288] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [CANDIDATE]: Term 1 election: Requested vote from peers 51926d98851c40f38b2828e2add3fc5e (127.0.98.1:40931), 576d5f8e17b1424a869b1f5400c39208 (127.0.98.3:35555)
I20260620 14:27:21.803953 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2809e174284487486908f4159d53768" candidate_uuid: "d35e9604e21b49c690e8497247e11ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576d5f8e17b1424a869b1f5400c39208"
I20260620 14:27:21.804066 12047 raft_consensus.cc:3037] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:27:21.805629 12047 raft_consensus.cc:2454] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35e9604e21b49c690e8497247e11ce1 in term 1.
I20260620 14:27:21.805969 11847 leader_election.cc:302] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576d5f8e17b1424a869b1f5400c39208, d35e9604e21b49c690e8497247e11ce1; no voters:
I20260620 14:27:21.806129 12118 raft_consensus.cc:2781] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:27:21.806329 12118 raft_consensus.cc:686] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 1 LEADER]: Becoming Leader. State: Replica: d35e9604e21b49c690e8497247e11ce1, State: Running, Role: LEADER
I20260620 14:27:21.806671 12118 consensus_queue.cc:237] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } } peers { permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 } } peers { permanent_uuid: "d35e9604e21b49c690e8497247e11ce1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37017 } }
I20260620 14:27:21.809957 11610 catalog_manager.cc:5495] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d35e9604e21b49c690e8497247e11ce1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d35e9604e21b49c690e8497247e11ce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d35e9604e21b49c690e8497247e11ce1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37017 } health_report { overall_health: HEALTHY } } }
I20260620 14:27:21.813603 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.793631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60865 (local address 127.0.98.1:40931)
0620 14:27:21.793732 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:21.793737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:21.799303 (+ 5566us) server_negotiation.cc:407] Connection header received
0620 14:27:21.799557 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:21.799560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:21.799612 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:21.799658 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:27:21.800083 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.800820 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.812335 (+ 11515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.812924 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.813012 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.813070 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:21.813419 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 14:27:21.813494 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:27:21.814111 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.793498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40931 (local address 127.0.98.2:60865)
0620 14:27:21.799216 (+ 5718us) negotiation.cc:107] Waiting for socket to connect
0620 14:27:21.799230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:27:21.799420 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:27:21.799792 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:27:21.799795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:27:21.799810 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:27:21.799956 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.799962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.811059 (+ 11097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.811062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.812195 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:27:21.812205 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.813155 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.813158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:27:21.813275 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.813294 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:27:21.813721 (+ 427us) client_negotiation.cc:240] Negotiation successful
0620 14:27:21.813791 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5657,"thread_start_us":89,"threads_started":1}
I20260620 14:27:21.814782 11776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2809e174284487486908f4159d53768" candidate_uuid: "d35e9604e21b49c690e8497247e11ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51926d98851c40f38b2828e2add3fc5e" is_pre_election: true
I20260620 14:27:21.815059 11776 raft_consensus.cc:2454] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d35e9604e21b49c690e8497247e11ce1 in term 0.
I20260620 14:27:21.815339 11775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2809e174284487486908f4159d53768" candidate_uuid: "d35e9604e21b49c690e8497247e11ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51926d98851c40f38b2828e2add3fc5e"
I20260620 14:27:21.815433 11775 raft_consensus.cc:3037] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:27:21.816614 11775 raft_consensus.cc:2454] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d35e9604e21b49c690e8497247e11ce1 in term 1.
W20260620 14:27:21.837339 12095 tablet.cc:2369] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:27:21.854053 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.849098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.0.98.2:37017)
0620 14:27:21.849494 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 14:27:21.849499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:27:21.849515 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:27:21.849597 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:27:21.849600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:27:21.849666 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:27:21.849774 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:27:21.850184 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.850936 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.852139 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:27:21.852471 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:27:21.852576 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:27:21.852951 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:27:21.853535 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:27:21.853648 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:27:21.853811 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:27:21.853874 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
Jun 20 14:27:21 dist-test-slave-pk3j krb5kdc[11134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:27:21.868644 11775 raft_consensus.cc:1261] T e2809e174284487486908f4159d53768 P 51926d98851c40f38b2828e2add3fc5e [term 1 FOLLOWER]: Refusing update from remote peer d35e9604e21b49c690e8497247e11ce1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:27:21.869725 12123 consensus_queue.cc:1035] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51926d98851c40f38b2828e2add3fc5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:27:21.872283 12047 raft_consensus.cc:1261] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208 [term 1 FOLLOWER]: Refusing update from remote peer d35e9604e21b49c690e8497247e11ce1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:27:21.873296 12118 consensus_queue.cc:1035] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "576d5f8e17b1424a869b1f5400c39208" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:27:21.890087 12127 mvcc.cc:205] Tried to move back new op lower bound from 7298931269085900800 to 7298931268843155456. Current Snapshot: MvccSnapshot[applied={T|T < 7298931269085900800}]
I20260620 14:27:21.894054 12130 mvcc.cc:205] Tried to move back new op lower bound from 7298931269085900800 to 7298931268843155456. Current Snapshot: MvccSnapshot[applied={T|T < 7298931269085900800}]
I20260620 14:27:21.908614 11610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:27:21.908779 11610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:27:21.911207 11610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32ae177b40d1420faa44157df247367a: Sending DeleteTablet for 3 replicas of tablet e2809e174284487486908f4159d53768
I20260620 14:27:21.912240 11756 tablet_service.cc:1514] Processing DeleteTablet for tablet e2809e174284487486908f4159d53768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:27:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38652
I20260620 14:27:21.912680 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet e2809e174284487486908f4159d53768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:27:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43240
I20260620 14:27:21.913388 12009 tablet_service.cc:1514] Processing DeleteTablet for tablet e2809e174284487486908f4159d53768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:27:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39466
I20260620 14:27:21.914333 12140 tablet_replica.cc:330] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: stopping tablet replica
I20260620 14:27:21.914460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 11691
I20260620 14:27:21.918774 12140 raft_consensus.cc:2229] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:27:21.919092 12140 raft_consensus.cc:2258] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:27:21.921370 12139 tablet_replica.cc:330] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: stopping tablet replica
I20260620 14:27:21.922250 12139 raft_consensus.cc:2229] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:27:21.922623 12139 raft_consensus.cc:2258] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:27:21.924116 12140 ts_tablet_manager.cc:1905] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:27:21.926815 12140 ts_tablet_manager.cc:1918] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:27:21.926916 12140 log.cc:1196] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/e2809e174284487486908f4159d53768
I20260620 14:27:21.927246 12140 ts_tablet_manager.cc:1939] T e2809e174284487486908f4159d53768 P 576d5f8e17b1424a869b1f5400c39208: Deleting consensus metadata
I20260620 14:27:21.928493 12139 ts_tablet_manager.cc:1905] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:27:21.934618 12139 ts_tablet_manager.cc:1918] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:27:21.934721 12139 log.cc:1196] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/e2809e174284487486908f4159d53768
I20260620 14:27:21.935057 12139 ts_tablet_manager.cc:1939] T e2809e174284487486908f4159d53768 P d35e9604e21b49c690e8497247e11ce1: Deleting consensus metadata
W20260620 14:27:21.935834 11597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40931 (error 108)
I20260620 14:27:21.937367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 11830
I20260620 14:27:21.944213 11594 catalog_manager.cc:4841] TS 576d5f8e17b1424a869b1f5400c39208 (127.0.98.3:35555): tablet e2809e174284487486908f4159d53768 (table test-table [id=25865e215b1c40519448456e9f8f75da]) successfully deleted
I20260620 14:27:21.945317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 11965
I20260620 14:27:21.946221 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.945769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40931 (local address 127.0.0.1:38658)
0620 14:27:21.946011 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 14:27:21.946105 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260620 14:27:21.946650 11597 catalog_manager.cc:4584] TS 51926d98851c40f38b2828e2add3fc5e (127.0.98.1:40931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2809e174284487486908f4159d53768: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40931: connect: Connection refused (error 111)
I20260620 14:27:21.948845 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:27:21.948599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37017 (local address 127.0.0.1:43262)
0620 14:27:21.948737 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:27:21.948783 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 14:27:21.953588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 11578
I20260620 14:27:21.961941 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:27:22.306370 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:27:22.306485 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:27:22.650831 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:27:22.651 UTC [11164] LOG: received smart shutdown request
2026-06-20 14:27:22.653 UTC [11164] LOG: background worker "logical replication launcher" (PID 11173) exited with exit code 1
2026-06-20 14:27:22.654 UTC [11168] LOG: shutting down
2026-06-20 14:27:22.816 UTC [11164] LOG: database system is shut down
2026-06-20T14:27:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42386 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:27:22 dist-test-slave-pk3j krb5kdc[12150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:27:22 dist-test-slave-pk3j krb5kdc[12150](info): set up 2 sockets
Jun 20 14:27:22 dist-test-slave-pk3j krb5kdc[12150](info): commencing operation
krb5kdc: starting...
W20260620 14:27:23.977087 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:27:24 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:27:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 14:27:24.127552 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDwUWGk/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/postgres -l logfile start
2026-06-20 14:27:26.369 UTC [12180] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:27:26.369 UTC [12180] LOG: listening on IPv4 address "127.0.98.20", port 41235
2026-06-20 14:27:26.372 UTC [12180] LOG: listening on Unix socket "/tmp/.s.PGSQL.41235"
2026-06-20 14:27:26.406 UTC [12183] LOG: database system was shut down at 2026-06-20 14:27:25 UTC
2026-06-20 14:27:26.409 UTC [12180] LOG: database system is ready to accept connections
I20260620 14:27:27.340729 392 mini_postgres.cc:96] Postgres bound to 41235
2026-06-20 14:27:27.346 UTC [12191] FATAL: database "slave" does not exist
127.0.98.20:41235 - accepting connections
I20260620 14:27:27.346980 392 mini_ranger.cc:161] Starting Ranger...
I20260620 14:27:27.360828 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:27:27.737879 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:27:27.737942 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-admin
2026-06-20 14:27:27,999 [I] DB FLAVOR :POSTGRES
2026-06-20 14:27:27,999 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:27:27,999 [I] Checking connection..
2026-06-20 14:27:28,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:27:28,249 [I] Checking connection passed.
2026-06-20 14:27:28,249 [I] --------- Verifying version history table ---------
2026-06-20 14:27:28,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:27:28,497 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:27:28,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:27:28,745 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:27:28,745 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:27:28,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:27:28.987 UTC [12255] WARNING: there is no transaction in progress
2026-06-20 14:27:28,993 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:27:28,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:27:29,211 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:27:29,211 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:27:29,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:27:29,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:27:29,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pk3j.c.gcp-upstream.internal','N') ;"
2026-06-20 14:27:29,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:27:30,141 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:27:30,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:27:30,396 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:27:30,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:27:30,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:27:30,885 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:27:30,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:27:31.748 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.763 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.778 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.791 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.808 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.826 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.876 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.882 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.895 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.904 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.913 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.925 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.934 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.940 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.946 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:31.953 UTC [12363] WARNING: there is no transaction in progress
2026-06-20 14:27:32,491 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:27:32,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:27:32,721 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:27:32,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:27:32,940 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:27:32,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:27:33,179 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:27:33,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:27:33,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:27:33,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pk3j.c.gcp-upstream.internal';"
2026-06-20 14:27:33,871 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:27:33,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:27:34,089 [I] DB_PATCHES have already been applied
I20260620 14:27:34.094664 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDwUWGk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:27:34.094719 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 14:27:34.097894 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:60651
Jun 20, 2026 2:27:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:27:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:27:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:27:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:27:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:27:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:27:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:27:35.203349 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.856s user 0.002s sys 0.006s
Jun 20, 2026 2:27:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:27:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:27:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:27:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:27:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:27:56.084352 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:27:56 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965676, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:27:56.171919 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:27:56.180270 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:27:56.866176 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:27:56.866257 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-kms
I20260620 14:27:56.866276 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41235
2026-06-20 14:27:57,071 [I] DB FLAVOR :POSTGRES
2026-06-20 14:27:57,071 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:27:57,071 [I] Checking connection
2026-06-20 14:27:57,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:27:57,297 [I] connection success
2026-06-20 14:27:57,298 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:27:57,298 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:27:57,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:27:57,517 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:27:57,517 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:27:57,518 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:27:57,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41235/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:27:57,757 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:27:58.192530 392 mini_ranger_kms.cc:310] Created kms service
I20260620 14:27:58.353778 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:27:58.394042 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:27:58 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965676, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:27:58.743279 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:27:58.743430 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDwUWGk/build/debug/bin/postgresql.jar:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDwUWGk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:27:58.743450 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 14:27:58.748776 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12543
I20260620 14:27:58.748852 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56445
Jun 20, 2026 2:27:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:27:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:27:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:27:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:27:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:27:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 14:28:00 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965680, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:00.957437 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.786s user 0.006s sys 0.181s
I20260620 14:28:00.957567 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:28:00.957650 392 mini_ranger_kms.cc:401] 127.0.98.20:56445/kms/v1/keys
Jun 20 14:28:01 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965680, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:28:01 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965680, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 14:28:03 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:28:03.550069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42503
--ranger_config_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56445/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:03.655546 12594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:03.655781 12594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:03.655819 12594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:03.659710 12594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:03.659760 12594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:03.659780 12594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:03.659795 12594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:03.659809 12594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:03.664974 12594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56445/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42503
--ranger_config_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:03.665943 12594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:03.666903 12594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:03.675045 12600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:03.675055 12599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:03.675055 12602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:03.675439 12594 server_base.cc:1034] running on GCE node
I20260620 14:28:03.676026 12594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:03.677318 12594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:03.678510 12594 hybrid_clock.cc:648] HybridClock initialized: now 1781965683678481 us; error 39 us; skew 500 ppm
Jun 20 14:28:03 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:03.685283 12594 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:03.686233 12594 webserver.cc:460] Webserver started at http://127.0.98.62:35599/ using document root <none> and password file <none>
I20260620 14:28:03.686774 12594 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:03.686873 12594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:03.687067 12594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:28:03 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:03.738242 12594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "4b393583dc3f4fd4994e67a3cdbcdf32"
format_stamp: "Formatted at 2026-06-20 14:28:03 on dist-test-slave-pk3j"
server_key: "66f201ef702a5f04c398094c900aba92"
server_key_iv: "38c9e195c08df3e9f49a71b4aef92b83"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:03.738839 12594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b393583dc3f4fd4994e67a3cdbcdf32"
format_stamp: "Formatted at 2026-06-20 14:28:03 on dist-test-slave-pk3j"
server_key: "66f201ef702a5f04c398094c900aba92"
server_key_iv: "38c9e195c08df3e9f49a71b4aef92b83"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:03.742663 12594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:28:03.755370 12609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:03.756696 12594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:28:03.756837 12594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "4b393583dc3f4fd4994e67a3cdbcdf32"
format_stamp: "Formatted at 2026-06-20 14:28:03 on dist-test-slave-pk3j"
server_key: "66f201ef702a5f04c398094c900aba92"
server_key_iv: "38c9e195c08df3e9f49a71b4aef92b83"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:03.756969 12594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:03.776216 12594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:03.779227 12594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:03.779426 12594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:03.787024 12594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42503
I20260620 14:28:03.787096 12661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42503 every 8 connection(s)
I20260620 14:28:03.787938 12594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:03.790613 12662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:03.795717 12662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: Bootstrap starting.
I20260620 14:28:03.796060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 12594
I20260620 14:28:03.796175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:28:03.796447 392 external_mini_cluster.cc:1428] Setting key 4cd82bc55a00752ee9b22366ba2090b8
I20260620 14:28:03.798976 12662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:03.799680 12662 log.cc:824] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:03.801750 12662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: No bootstrap required, opened a new log
I20260620 14:28:03.805745 12662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 14:28:03.805971 12662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:03.806057 12662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b393583dc3f4fd4994e67a3cdbcdf32, State: Initialized, Role: FOLLOWER
Jun 20 14:28:03 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:03.807123 12662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 14:28:03.807302 12662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:03.807425 12662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:03.807535 12662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:03.808656 12662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 14:28:03.809011 12662 leader_election.cc:302] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b393583dc3f4fd4994e67a3cdbcdf32; no voters:
I20260620 14:28:03.809340 12662 leader_election.cc:288] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:28:03.809443 12667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:03.809708 12667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 1 LEADER]: Becoming Leader. State: Replica: 4b393583dc3f4fd4994e67a3cdbcdf32, State: Running, Role: LEADER
I20260620 14:28:03.810066 12667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 14:28:03.810699 12662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:03.811321 12667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } } }
I20260620 14:28:03.811514 12667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:03.811753 12667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b393583dc3f4fd4994e67a3cdbcdf32. Latest consensus state: current_term: 1 leader_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } } }
I20260620 14:28:03.811887 12667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:03.813066 12662 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:28:03.813297 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:03.797782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.0.98.62:42503)
0620 14:28:03.798183 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:03.798203 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:03.798246 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:28:03.799067 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:03.799095 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:03.799533 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:03.799887 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:03.800903 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:03.802469 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:03.803685 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:03.804382 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:03.804535 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:03.807402 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:03.807407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:03.807423 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:03.807453 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:03.809492 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:03.809994 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:03.810000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:03.810005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:03.810115 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:03.810397 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:03.810400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:03.810401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:03.810919 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:03.811108 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:03.811958 (+ 850us) server_negotiation.cc:299] Negotiation successful
0620 14:28:03.812215 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":91,"threads_started":1}
I20260620 14:28:03.818482 12675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:03.821982 12675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:03.828509 12675 catalog_manager.cc:1347] Generated new cluster ID: 704e64eaeebd4a539a3277efb83bbe92
I20260620 14:28:03.828559 12675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:03.847493 12675 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:28:03.848630 12675 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:03.853502 12675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: Generated new TSK 0
I20260620 14:28:03.854218 12675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:28:04.547824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:04.714435 12707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:04.714838 12707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:04.714954 12707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:04.720820 12707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:04.720935 12707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:04.721065 12707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:04.728344 12707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:04.729774 12707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:04.730754 12707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:04.739857 12715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:04.740320 12717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:04.740404 12707 server_base.cc:1034] running on GCE node
W20260620 14:28:04.740546 12714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:04.740954 12707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:04.741641 12707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:04.742794 12707 hybrid_clock.cc:648] HybridClock initialized: now 1781965684742767 us; error 39 us; skew 500 ppm
Jun 20 14:28:04 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:04.745923 12707 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:04.746842 12707 webserver.cc:460] Webserver started at http://127.0.98.1:38501/ using document root <none> and password file <none>
I20260620 14:28:04.747445 12707 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:04.747510 12707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:04.747702 12707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:28:04 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:04.763156 12707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "4d37af67f216464b9adfd8b014761a13"
format_stamp: "Formatted at 2026-06-20 14:28:04 on dist-test-slave-pk3j"
server_key: "ba6ff859659d8586b42824729c911112"
server_key_iv: "4983a5adac8da5e75427ef16ace1a8c3"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:04.763830 12707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4d37af67f216464b9adfd8b014761a13"
format_stamp: "Formatted at 2026-06-20 14:28:04 on dist-test-slave-pk3j"
server_key: "ba6ff859659d8586b42824729c911112"
server_key_iv: "4983a5adac8da5e75427ef16ace1a8c3"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:04.771139 12707 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 14:28:04.785933 12726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:04.787499 12707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:04.787663 12707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "4d37af67f216464b9adfd8b014761a13"
format_stamp: "Formatted at 2026-06-20 14:28:04 on dist-test-slave-pk3j"
server_key: "ba6ff859659d8586b42824729c911112"
server_key_iv: "4983a5adac8da5e75427ef16ace1a8c3"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:04.787860 12707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:04.807507 12707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:04.814951 12707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:04.815138 12707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:04.815755 12707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:04.816723 12707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:04.816799 12707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:04.816861 12707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:04.816887 12707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:04.832386 12707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35435
I20260620 14:28:04.833460 12707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:04.837692 12839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35435 every 8 connection(s)
I20260620 14:28:04.840580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 12707
I20260620 14:28:04.840687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:28:04.840986 392 external_mini_cluster.cc:1428] Setting key 9045d2734fb7afac9e020e58b6bb3b38
Jun 20 14:28:04 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:04.851244 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:04.837438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36927 (local address 127.0.98.62:42503)
0620 14:28:04.839452 (+ 2014us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:04.839461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:04.839475 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:28:04.839517 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:04.839520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:04.839566 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:04.839649 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:04.841053 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:04.842332 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:04.843949 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:04.844320 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:04.844420 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:04.847202 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:04.847205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:04.847208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:04.847230 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:04.849006 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:04.849578 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:04.849581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:04.849583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:04.849664 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:04.850006 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:04.850009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:04.850011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:04.850201 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:04.850284 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:04.850907 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 14:28:04.851030 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1936,"thread_start_us":83,"threads_started":1}
I20260620 14:28:04.852725 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:04.836390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42503 (local address 127.0.98.1:36927)
0620 14:28:04.838024 (+ 1634us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:04.838073 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:04.839043 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:04.839929 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:04.839938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:04.840254 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:04.840872 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:04.840907 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:04.842479 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:04.842483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:04.843815 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:04.843827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:04.844584 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:04.844588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:04.844767 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:04.845404 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:04.845423 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:04.847029 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:04.849129 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:04.849134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:04.849147 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:04.849470 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:04.849754 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:04.849757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:04.849759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:04.849908 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:04.850343 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:04.850349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:04.850615 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 14:28:04.851431 (+ 816us) client_negotiation.cc:240] Negotiation successful
0620 14:28:04.851741 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"thread_start_us":130,"threads_started":1}
I20260620 14:28:04.853986 12840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42503
I20260620 14:28:04.854212 12840 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:04.854725 12840 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 14:28:04.856338 12626 ts_manager.cc:194] Registered new tserver with Master: 4d37af67f216464b9adfd8b014761a13 (127.0.98.1:35435)
I20260620 14:28:04.857800 12626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36927
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 20 14:28:04 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:28:04.881640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:05.030100 12846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:05.030481 12846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:05.030588 12846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:05.036641 12846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:05.036710 12846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:05.036792 12846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:05.044073 12846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:05.045234 12846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:05.046448 12846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:05.057977 12852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:05.057979 12853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:05.058657 12855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:05.059574 12846 server_base.cc:1034] running on GCE node
I20260620 14:28:05.059965 12846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:05.060648 12846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:05.061836 12846 hybrid_clock.cc:648] HybridClock initialized: now 1781965685061819 us; error 30 us; skew 500 ppm
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:05.065069 12846 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:05.066005 12846 webserver.cc:460] Webserver started at http://127.0.98.2:37623/ using document root <none> and password file <none>
I20260620 14:28:05.066633 12846 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:05.066700 12846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:05.066910 12846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:05.080004 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "ab0a0c44f5994460a1ab30e5b073ec7f"
format_stamp: "Formatted at 2026-06-20 14:28:05 on dist-test-slave-pk3j"
server_key: "b7e0898e22b9437ebd93fba3a0175652"
server_key_iv: "7d98b95f7dbf2b0344b396238599be30"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:05.080718 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ab0a0c44f5994460a1ab30e5b073ec7f"
format_stamp: "Formatted at 2026-06-20 14:28:05 on dist-test-slave-pk3j"
server_key: "b7e0898e22b9437ebd93fba3a0175652"
server_key_iv: "7d98b95f7dbf2b0344b396238599be30"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:05.085471 12846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:28:05.100555 12862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:05.102169 12846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:05.102308 12846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "ab0a0c44f5994460a1ab30e5b073ec7f"
format_stamp: "Formatted at 2026-06-20 14:28:05 on dist-test-slave-pk3j"
server_key: "b7e0898e22b9437ebd93fba3a0175652"
server_key_iv: "7d98b95f7dbf2b0344b396238599be30"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:05.102429 12846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:05.117550 12846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:05.120345 12846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:05.120523 12846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:05.121048 12846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:05.121901 12846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:05.121944 12846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:05.121981 12846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:05.122036 12846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:05.132058 12846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36509
I20260620 14:28:05.133064 12846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:05.135310 12975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36509 every 8 connection(s)
I20260620 14:28:05.141930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 12846
I20260620 14:28:05.142041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:28:05.142347 392 external_mini_cluster.cc:1428] Setting key 9dcaa3a40893695497b9d1898a3d7c78
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:28:05.159351 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:05.137374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53605 (local address 127.0.98.62:42503)
0620 14:28:05.137521 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:05.137526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:05.138630 (+ 1104us) server_negotiation.cc:407] Connection header received
0620 14:28:05.139806 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:05.139811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:05.139869 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:05.139946 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:05.143723 (+ 3777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:05.144589 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:05.148629 (+ 4040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:05.148982 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:05.149059 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:05.155173 (+ 6114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:05.155177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:05.155180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:05.155207 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:05.156979 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:05.157586 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:05.157590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:05.157592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:05.157674 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:05.158064 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:05.158068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:05.158070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:05.158270 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:05.158360 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:05.159022 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 14:28:05.159151 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:28:05.161036 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:05.137845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42503 (local address 127.0.98.2:53605)
0620 14:28:05.138476 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:05.138528 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:05.139556 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:05.141484 (+ 1928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:05.141494 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:05.142934 (+ 1440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:05.143544 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:05.143565 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:05.144745 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:05.144759 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:05.148491 (+ 3732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:05.148502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:05.149275 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:05.149279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:05.149492 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:05.150162 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:05.150182 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:05.154991 (+ 4809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:05.157108 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:05.157114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:05.157128 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:05.157469 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:05.157791 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:05.157794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:05.157796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:05.157954 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:05.158425 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:05.158431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:05.158712 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 14:28:05.159555 (+ 843us) client_negotiation.cc:240] Negotiation successful
0620 14:28:05.159862 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":128,"threads_started":1}
I20260620 14:28:05.162166 12976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42503
I20260620 14:28:05.162391 12976 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:05.162892 12976 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 14:28:05.163893 12626 ts_manager.cc:194] Registered new tserver with Master: ab0a0c44f5994460a1ab30e5b073ec7f (127.0.98.2:36509)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:28:05.164592 12626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:28:05.180419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 14:28:05.331120 12981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:05.331365 12981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:05.331439 12981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:05.336125 12981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:05.336186 12981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:05.336284 12981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:05.342967 12981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:05.344128 12981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:05.345166 12981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:05.357872 12987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:05.358180 12986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:05.358680 12989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:05.361579 12981 server_base.cc:1034] running on GCE node
I20260620 14:28:05.362114 12981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:05.362813 12981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:05.364005 12981 hybrid_clock.cc:648] HybridClock initialized: now 1781965685363946 us; error 73 us; skew 500 ppm
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:05.367390 12981 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:05.368472 12981 webserver.cc:460] Webserver started at http://127.0.98.3:44703/ using document root <none> and password file <none>
I20260620 14:28:05.369115 12981 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:05.369189 12981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:05.369431 12981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:05.384083 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "d52656129db14ff6a1d00209611b014c"
format_stamp: "Formatted at 2026-06-20 14:28:05 on dist-test-slave-pk3j"
server_key: "1cfb6c063ec8bd481f67ef703bf79151"
server_key_iv: "30d3c8c43db7e35ed638aa795d38025e"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:05.384837 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d52656129db14ff6a1d00209611b014c"
format_stamp: "Formatted at 2026-06-20 14:28:05 on dist-test-slave-pk3j"
server_key: "1cfb6c063ec8bd481f67ef703bf79151"
server_key_iv: "30d3c8c43db7e35ed638aa795d38025e"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:05.394134 12981 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260620 14:28:05.408586 12996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:05.410027 12981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:28:05.410141 12981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "d52656129db14ff6a1d00209611b014c"
format_stamp: "Formatted at 2026-06-20 14:28:05 on dist-test-slave-pk3j"
server_key: "1cfb6c063ec8bd481f67ef703bf79151"
server_key_iv: "30d3c8c43db7e35ed638aa795d38025e"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:05.410254 12981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:05.439530 12981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:05.442926 12981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:05.443102 12981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:05.443738 12981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:05.444717 12981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:05.444785 12981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:05.444836 12981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:05.444864 12981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:05.456063 12981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39247
I20260620 14:28:05.456265 13109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39247 every 8 connection(s)
I20260620 14:28:05.457114 12981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:28:05.457911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 12981
I20260620 14:28:05.458029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:28:05.458300 392 external_mini_cluster.cc:1428] Setting key 36d1462c14e29762354dc55a11ddbb7b
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:05.474139 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:05.460025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50275 (local address 127.0.98.62:42503)
0620 14:28:05.460141 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:05.460145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:05.461172 (+ 1027us) server_negotiation.cc:407] Connection header received
0620 14:28:05.462325 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:05.462328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:05.462376 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:05.462461 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:05.463719 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:05.464499 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:05.466186 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:05.466504 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:05.466605 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:05.469754 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:05.469758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:05.469761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:05.469794 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:05.471514 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:05.472297 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:05.472300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:05.472302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:05.472380 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:05.472742 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:05.472747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:05.472750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:05.473097 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:05.473178 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:05.473877 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 14:28:05.473986 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:28:05.475068 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:05.460490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42503 (local address 127.0.98.3:50275)
0620 14:28:05.461021 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:05.461074 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:05.462107 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:05.462620 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:05.462629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:05.462948 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:05.463576 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:05.463595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:05.464642 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:05.464646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:05.466060 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:05.466071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:05.466664 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:05.466668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:05.466882 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:05.467470 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:05.467490 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:05.469592 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:05.471858 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:05.471863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:05.471876 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:05.472188 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:05.472477 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:05.472480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:05.472481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:05.472631 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:05.473267 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:05.473273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:05.473537 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 14:28:05.473793 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 14:28:05.474063 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":137,"threads_started":1}
I20260620 14:28:05.476466 13110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42503
I20260620 14:28:05.476691 13110 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:05.477277 13110 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 14:28:05.478288 12626 ts_manager.cc:194] Registered new tserver with Master: d52656129db14ff6a1d00209611b014c (127.0.98.3:39247)
I20260620 14:28:05.478963 12626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50275
I20260620 14:28:05.485164 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:28:05.485982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 12707
I20260620 14:28:05.493276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 12846
I20260620 14:28:05.498939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 12981
I20260620 14:28:05.504112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 12594
I20260620 14:28:05.510911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42503
--webserver_interface=127.0.98.62
--webserver_port=35599
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42503
--ranger_config_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56445/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56445/kms with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:05.613694 13113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:05.613948 13113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:05.614024 13113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:05.617865 13113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:05.617941 13113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:05.617978 13113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:05.618032 13113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:05.618067 13113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:05.622884 13113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56445/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42503
--ranger_config_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:05.623697 13113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:05.624540 13113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:05.632161 13119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:05.632166 13118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:05.632365 13121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:05.632846 13113 server_base.cc:1034] running on GCE node
I20260620 14:28:05.633437 13113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:05.634577 13113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:05.635736 13113 hybrid_clock.cc:648] HybridClock initialized: now 1781965685635723 us; error 30 us; skew 500 ppm
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:05.638231 13113 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:05.638993 13113 webserver.cc:460] Webserver started at http://127.0.98.62:35599/ using document root <none> and password file <none>
I20260620 14:28:05.639470 13113 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:05.639554 13113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:05.684959 13113 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:28:05.687304 13129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:05.688571 13113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:28:05.688706 13113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "4b393583dc3f4fd4994e67a3cdbcdf32"
format_stamp: "Formatted at 2026-06-20 14:28:03 on dist-test-slave-pk3j"
server_key: "66f201ef702a5f04c398094c900aba92"
server_key_iv: "38c9e195c08df3e9f49a71b4aef92b83"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:05.689240 13113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:05.715255 13113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:05.718256 13113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:05.718463 13113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:05.726069 13113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42503
I20260620 14:28:05.726087 13181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42503 every 8 connection(s)
I20260620 14:28:05.727154 13113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:05.727334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 13113
I20260620 14:28:05.727862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35435
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42503
--webserver_port=38501
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56445/kms with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:05.732594 13182 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:28:05.734695 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: Bootstrap starting.
I20260620 14:28:05.755503 13182 log.cc:824] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:05.763309 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:05.763715 13182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: Bootstrap complete.
I20260620 14:28:05.769338 13182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 14:28:05.769809 13182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b393583dc3f4fd4994e67a3cdbcdf32, State: Initialized, Role: FOLLOWER
I20260620 14:28:05.770319 13182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 14:28:05.770463 13182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:05.770526 13182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:05.770608 13182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:28:05.772998 13182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 14:28:05.773363 13182 leader_election.cc:302] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b393583dc3f4fd4994e67a3cdbcdf32; no voters:
I20260620 14:28:05.773691 13182 leader_election.cc:288] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:28:05.773859 13186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:28:05.774124 13186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [term 2 LEADER]: Becoming Leader. State: Replica: 4b393583dc3f4fd4994e67a3cdbcdf32, State: Running, Role: LEADER
I20260620 14:28:05.774508 13186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } }
I20260620 14:28:05.774883 13182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:05.775460 13186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } } }
I20260620 14:28:05.775624 13186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:05.775871 13186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b393583dc3f4fd4994e67a3cdbcdf32. Latest consensus state: current_term: 2 leader_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b393583dc3f4fd4994e67a3cdbcdf32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42503 } } }
I20260620 14:28:05.775993 13186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:05.776247 13194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:05.779188 13194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:05.779624 13182 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:28:05.779910 13194 catalog_manager.cc:1259] Loaded cluster ID: 704e64eaeebd4a539a3277efb83bbe92
I20260620 14:28:05.780000 13194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:05.782284 13194 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:05.782923 13194 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: Loaded TSK: 0
I20260620 14:28:05.784046 13194 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:28:05.864756 13184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:05.865043 13184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:05.865098 13184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:05.870968 13184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:05.871034 13184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:05.871110 13184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:05.878252 13184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38501
--tserver_master_addrs=127.0.98.62:42503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:05.879340 13184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:05.880465 13184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:05.891499 13219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:05.892423 13222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:05.892716 13220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:05.893404 13184 server_base.cc:1034] running on GCE node
I20260620 14:28:05.893797 13184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:05.894434 13184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:05.897078 13184 hybrid_clock.cc:648] HybridClock initialized: now 1781965685897048 us; error 40 us; skew 500 ppm
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:05.900357 13184 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:05.901301 13184 webserver.cc:460] Webserver started at http://127.0.98.1:38501/ using document root <none> and password file <none>
I20260620 14:28:05.901911 13184 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:05.901988 13184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:28:05 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:05.930420 13184 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260620 14:28:05.933452 13230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:05.934929 13184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:05.935060 13184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "4d37af67f216464b9adfd8b014761a13"
format_stamp: "Formatted at 2026-06-20 14:28:04 on dist-test-slave-pk3j"
server_key: "ba6ff859659d8586b42824729c911112"
server_key_iv: "4983a5adac8da5e75427ef16ace1a8c3"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:05.935801 13184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:05.987807 13184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:05.991328 13184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:05.991518 13184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:05.992268 13184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:05.993256 13184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:05.993312 13184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:05.993366 13184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:05.993392 13184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:06.003536 13184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35435
I20260620 14:28:06.003654 13343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35435 every 8 connection(s)
I20260620 14:28:06.004808 13184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:06.008448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 13184
I20260620 14:28:06.008904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36509
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42503
--webserver_port=37623
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56445/kms with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:28:06 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:06.024857 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:06.007208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42503 (local address 127.0.98.1:55333)
0620 14:28:06.008440 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:06.008497 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:06.009677 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:06.011107 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:06.011115 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:06.011436 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:06.012026 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:06.012045 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.013912 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.013929 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:06.015237 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:06.015248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.016436 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.016440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:06.016617 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:06.017282 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:06.017301 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:06.018984 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:06.021398 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:06.021403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:06.021416 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:06.021743 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:06.022136 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:06.022139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:06.022142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:06.022312 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:06.023079 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:06.023084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:06.023383 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 14:28:06.023587 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 14:28:06.023887 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"thread_start_us":118,"threads_started":1}
I20260620 14:28:06.026070 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:06.007026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55333 (local address 127.0.98.62:42503)
0620 14:28:06.008140 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:06.008161 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:06.009984 (+ 1823us) server_negotiation.cc:407] Connection header received
0620 14:28:06.010282 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:06.010299 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:06.010688 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:06.011047 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:06.012219 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.013757 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.015388 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.016063 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.016214 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:06.019188 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:06.019194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:06.019210 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:06.019239 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:06.021252 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:06.021880 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:06.021886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:06.021891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:06.022026 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:06.022442 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:06.022446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:06.022447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:06.022894 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:06.023023 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:06.025102 (+ 2079us) server_negotiation.cc:299] Negotiation successful
0620 14:28:06.025392 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":110,"threads_started":1}
I20260620 14:28:06.028288 13344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42503
I20260620 14:28:06.028512 13344 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:06.029052 13344 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
W20260620 14:28:06.031476 13344 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42503 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:28:06.031621 13344 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:06.031816 13344 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 14:28:06.032832 13344 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:06.033010 13344 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
W20260620 14:28:06.033901 13344 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:28:06.167797 13348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:06.168128 13348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:06.168193 13348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:06.173864 13348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:06.173931 13348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:06.174005 13348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:06.181337 13348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37623
--tserver_master_addrs=127.0.98.62:42503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:06.182502 13348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:06.183714 13348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:06.193704 13353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:06.193858 13356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:06.193706 13354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:06.194058 13348 server_base.cc:1034] running on GCE node
I20260620 14:28:06.194501 13348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:06.195168 13348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:06.196373 13348 hybrid_clock.cc:648] HybridClock initialized: now 1781965686196345 us; error 45 us; skew 500 ppm
Jun 20 14:28:06 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:06.199753 13348 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:06.200732 13348 webserver.cc:460] Webserver started at http://127.0.98.2:37623/ using document root <none> and password file <none>
I20260620 14:28:06.201375 13348 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:06.201476 13348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:28:06 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:06.228250 13348 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:06.231156 13364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:06.232601 13348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:28:06.232748 13348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "ab0a0c44f5994460a1ab30e5b073ec7f"
format_stamp: "Formatted at 2026-06-20 14:28:05 on dist-test-slave-pk3j"
server_key: "b7e0898e22b9437ebd93fba3a0175652"
server_key_iv: "7d98b95f7dbf2b0344b396238599be30"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:06.233330 13348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:06.287232 13348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:06.290831 13348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:06.291029 13348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:06.291671 13348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:06.292689 13348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:06.292745 13348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:06.292800 13348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:06.292829 13348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:06.303486 13348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36509
I20260620 14:28:06.304638 13348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:06.305753 13477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36509 every 8 connection(s)
I20260620 14:28:06.312930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 13348
I20260620 14:28:06.313434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39247
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42503
--webserver_port=44703
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56445/kms with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:28:06 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:06.327214 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:06.307979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42503 (local address 127.0.98.2:56143)
0620 14:28:06.313396 (+ 5417us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:06.313449 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:06.314679 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:06.315217 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:06.315226 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:06.315571 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:06.316201 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:06.316219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.317404 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.317407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:06.318784 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:06.318795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.319570 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.319574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:06.319756 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:06.320388 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:06.320406 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:06.322086 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:06.324163 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:06.324169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:06.324184 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:06.324496 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:06.324796 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:06.324799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:06.324801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:06.324965 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:06.325466 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:06.325472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:06.325772 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 14:28:06.325973 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 14:28:06.326256 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5180,"thread_start_us":133,"threads_started":1}
I20260620 14:28:06.327832 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:06.307465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56143 (local address 127.0.98.62:42503)
0620 14:28:06.307586 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:06.307591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:06.313552 (+ 5961us) server_negotiation.cc:407] Connection header received
0620 14:28:06.314915 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:06.314919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:06.314978 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:06.315055 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:06.316370 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.317234 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.318935 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.319388 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.319531 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:06.322279 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:06.322283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:06.322286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:06.322314 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:06.324035 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:06.324620 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:06.324623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:06.324625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:06.324705 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:06.325089 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:06.325092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:06.325094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:06.325320 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:06.325413 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:06.327568 (+ 2155us) server_negotiation.cc:299] Negotiation successful
0620 14:28:06.327687 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:28:06.329515 13478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42503
I20260620 14:28:06.329741 13478 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:06.330277 13478 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
W20260620 14:28:06.331947 13478 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42503 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:28:06.332065 13478 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:06.332217 13478 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 14:28:06.333077 13478 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:06.333267 13478 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
W20260620 14:28:06.334097 13478 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:28:06.444852 13481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:06.445115 13481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:06.445186 13481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:06.450251 13481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:06.450317 13481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:06.450405 13481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:06.456878 13481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44703
--tserver_master_addrs=127.0.98.62:42503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:06.458036 13481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:06.459244 13481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:06.468108 13488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:06.468119 13487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:06.468518 13490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:06.469050 13481 server_base.cc:1034] running on GCE node
I20260620 14:28:06.469437 13481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:06.470079 13481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:06.471261 13481 hybrid_clock.cc:648] HybridClock initialized: now 1781965686471234 us; error 41 us; skew 500 ppm
Jun 20 14:28:06 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:06.474751 13481 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:06.475684 13481 webserver.cc:460] Webserver started at http://127.0.98.3:44703/ using document root <none> and password file <none>
I20260620 14:28:06.476303 13481 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:06.476375 13481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:28:06 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:06.491245 13481 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:28:06.493958 13498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:06.496302 13481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:28:06.496416 13481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "d52656129db14ff6a1d00209611b014c"
format_stamp: "Formatted at 2026-06-20 14:28:05 on dist-test-slave-pk3j"
server_key: "1cfb6c063ec8bd481f67ef703bf79151"
server_key_iv: "30d3c8c43db7e35ed638aa795d38025e"
server_key_version: "kuduclusterkey@0"
I20260620 14:28:06.497088 13481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:06.516173 13481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:06.519475 13481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:06.519680 13481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:06.520255 13481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:06.521185 13481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:06.521288 13481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:06.521346 13481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:06.521373 13481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:06.531886 13481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39247
I20260620 14:28:06.532012 13611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39247 every 8 connection(s)
I20260620 14:28:06.532992 13481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:28:06.542222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 13481
Jun 20 14:28:06 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:06.550541 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:06.536087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56141 (local address 127.0.98.62:42503)
0620 14:28:06.536234 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:06.536239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:06.537172 (+ 933us) server_negotiation.cc:407] Connection header received
0620 14:28:06.538420 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:06.538424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:06.538480 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:06.538534 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:06.540051 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.540799 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.543051 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.543385 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.543478 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:06.546268 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:06.546272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:06.546275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:06.546302 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:06.548160 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:06.548756 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:06.548760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:06.548762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:06.548860 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:06.549327 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:06.549331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:06.549333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:06.549538 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:06.549622 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:06.550264 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 14:28:06.550382 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:28:06.552114 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:06.536482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42503 (local address 127.0.98.3:56141)
0620 14:28:06.537002 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:06.537059 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:06.538175 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:06.538930 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:06.538940 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:06.539235 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:06.539862 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:06.539884 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.540973 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.540976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:06.542912 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:06.542923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.543664 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.543668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:06.543852 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:06.544436 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:06.544454 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:06.546081 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:06.548318 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:06.548324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:06.548340 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:06.548644 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:06.549016 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:06.549019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:06.549021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:06.549183 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:06.549688 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:06.549693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:06.549996 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 14:28:06.550943 (+ 947us) client_negotiation.cc:240] Negotiation successful
0620 14:28:06.551229 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":144,"threads_started":1}
I20260620 14:28:06.554144 13612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42503
I20260620 14:28:06.554402 13612 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:06.554962 13612 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
W20260620 14:28:06.556488 13612 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42503 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:28:06.556635 13612 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:06.556802 13612 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 14:28:06.557569 13612 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:06.557740 13612 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
W20260620 14:28:06.558531 13612 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:28:06.558796 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:06.543188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.0.98.62:42503)
0620 14:28:06.549284 (+ 6096us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:06.549293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:06.549352 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:28:06.549388 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:06.549391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:06.549436 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:06.549519 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:06.550613 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.551402 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.552603 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:06.552954 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:06.553058 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:06.554747 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:06.554750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:06.554753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:06.554776 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:06.556553 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:06.557155 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:06.557159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:06.557161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:06.557267 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:06.557955 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:06.557958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:06.557960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:06.558167 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:06.558375 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:06.558506 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:28:06.558624 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6030,"thread_start_us":69,"threads_started":1}
Jun 20 14:28:06 dist-test-slave-pk3j krb5kdc[12150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781965686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:07.034124 13344 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:07.034402 13344 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 14:28:07.035512 13146 ts_manager.cc:194] Registered new tserver with Master: 4d37af67f216464b9adfd8b014761a13 (127.0.98.1:35435)
I20260620 14:28:07.037166 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55333
Jun 20 14:28:07 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:07.334335 13478 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:07.334620 13478 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 14:28:07.335381 13146 ts_manager.cc:194] Registered new tserver with Master: ab0a0c44f5994460a1ab30e5b073ec7f (127.0.98.2:36509)
I20260620 14:28:07.336143 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56143
Jun 20 14:28:07 dist-test-slave-pk3j krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 14:28:07.558817 13612 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:07.559095 13612 heartbeater.cc:511] Master 127.0.98.62:42503 requested a full tablet report, sending...
I20260620 14:28:07.560046 13146 ts_manager.cc:194] Registered new tserver with Master: d52656129db14ff6a1d00209611b014c (127.0.98.3:39247)
I20260620 14:28:07.560797 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56141
I20260620 14:28:07.567806 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:28:07.581487 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.571273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.0.98.62:42503)
0620 14:28:07.571848 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:07.571853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:07.572184 (+ 331us) server_negotiation.cc:407] Connection header received
0620 14:28:07.572222 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:07.572225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:07.572298 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:07.572376 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:07.573337 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.574171 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.575382 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.575722 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.575814 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.577679 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:07.577683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:07.577685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:07.577709 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:07.579434 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:07.579889 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:07.579893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:07.579895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:07.579976 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:07.580248 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:07.580252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:07.580253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:07.580460 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:07.580579 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:07.581166 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 14:28:07.581322 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":77,"threads_started":1}
I20260620 14:28:07.585167 13146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:28:07.589956 13146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:28:07.612345 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.604704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36509 (local address 127.0.0.1:40584)
0620 14:28:07.605421 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:07.605433 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:07.605502 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:07.606320 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:07.606323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:07.606337 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:07.606428 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:07.606434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.608100 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.608103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:07.610639 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:07.610648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.611798 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.611801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:07.611903 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.611954 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:28:07.612058 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:28:07.612173 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":1258,"thread_start_us":71,"threads_started":1}
I20260620 14:28:07.612792 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.602769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39247 (local address 127.0.0.1:55194)
0620 14:28:07.604800 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:07.604813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:07.604897 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:07.605613 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:07.605618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:07.605639 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:07.605756 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:07.605763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.608493 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.608496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:07.610369 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:07.610379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.611700 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.611704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:07.612502 (+ 798us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.612525 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:28:07.612604 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:28:07.612676 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1961,"thread_start_us":66,"threads_started":1}
I20260620 14:28:07.613307 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.603343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.0.98.3:39247)
0620 14:28:07.604234 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:07.604242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:07.604985 (+ 743us) server_negotiation.cc:407] Connection header received
0620 14:28:07.605061 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:07.605066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:07.605278 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:07.605432 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:07.607262 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.608349 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.610550 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.611384 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.611486 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.611639 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:07.613059 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0620 14:28:07.613153 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":105,"threads_started":1}
I20260620 14:28:07.615016 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.602661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35435 (local address 127.0.0.1:56828)
0620 14:28:07.603803 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:07.603850 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:07.603974 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:07.607112 (+ 3138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:07.607115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:07.607131 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:07.607209 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:07.607217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.612692 (+ 5475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.612695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:07.613776 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:07.613786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.614629 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.614632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:07.614741 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.614760 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:28:07.614841 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:28:07.614904 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":84,"threads_started":1}
I20260620 14:28:07.615078 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.605079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40584 (local address 127.0.98.2:36509)
0620 14:28:07.605711 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:07.605717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:07.605738 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:28:07.605806 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:07.605814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:07.606037 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:07.606171 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:07.606553 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.607975 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.610793 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.611541 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.611629 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.611747 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:07.614862 (+ 3115us) server_negotiation.cc:299] Negotiation successful
0620 14:28:07.614949 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":74,"threads_started":1}
I20260620 14:28:07.615520 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.603483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.0.98.1:35435)
0620 14:28:07.606721 (+ 3238us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:07.606727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:07.606748 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:28:07.606815 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:07.606820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:07.606990 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:07.607123 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:07.608628 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.609751 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.613926 (+ 4175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.614500 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.614629 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.614776 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:07.615287 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 14:28:07.615393 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3141,"thread_start_us":107,"threads_started":1}
I20260620 14:28:07.616302 13538 tablet_service.cc:1467] Processing CreateTablet for tablet fc97341efda14c7ebc7bd1332078d90f (DEFAULT_TABLE table=test-table [id=4f101d85747f42c1b61e12bbd9f22a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:07.617447 13538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc97341efda14c7ebc7bd1332078d90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:07.617734 13398 tablet_service.cc:1467] Processing CreateTablet for tablet fc97341efda14c7ebc7bd1332078d90f (DEFAULT_TABLE table=test-table [id=4f101d85747f42c1b61e12bbd9f22a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:07.618489 13398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc97341efda14c7ebc7bd1332078d90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:07.623003 13278 tablet_service.cc:1467] Processing CreateTablet for tablet fc97341efda14c7ebc7bd1332078d90f (DEFAULT_TABLE table=test-table [id=4f101d85747f42c1b61e12bbd9f22a7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:07.623907 13278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc97341efda14c7ebc7bd1332078d90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:07.625108 13648 tablet_bootstrap.cc:492] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: Bootstrap starting.
I20260620 14:28:07.627383 13648 tablet_bootstrap.cc:654] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:07.628046 13648 log.cc:824] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:07.630021 13648 tablet_bootstrap.cc:492] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: No bootstrap required, opened a new log
I20260620 14:28:07.630285 13648 ts_tablet_manager.cc:1397] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:28:07.632107 13650 tablet_bootstrap.cc:492] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13: Bootstrap starting.
I20260620 14:28:07.634147 13650 tablet_bootstrap.cc:654] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:07.634977 13650 log.cc:824] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:07.635263 13648 raft_consensus.cc:348] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d37af67f216464b9adfd8b014761a13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35435 } } peers { permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 } } peers { permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 } }
I20260620 14:28:07.635566 13648 raft_consensus.cc:374] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:07.635560 13651 tablet_bootstrap.cc:492] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: Bootstrap starting.
I20260620 14:28:07.635659 13648 raft_consensus.cc:729] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0a0c44f5994460a1ab30e5b073ec7f, State: Initialized, Role: FOLLOWER
I20260620 14:28:07.636166 13648 consensus_queue.cc:260] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d37af67f216464b9adfd8b014761a13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35435 } } peers { permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 } } peers { permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 } }
I20260620 14:28:07.637315 13648 ts_tablet_manager.cc:1428] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:28:07.637717 13478 heartbeater.cc:503] Master 127.0.98.62:42503 was elected leader, sending a full tablet report...
I20260620 14:28:07.638049 13651 tablet_bootstrap.cc:654] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:07.641312 13651 log.cc:824] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:07.643174 13650 tablet_bootstrap.cc:492] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13: No bootstrap required, opened a new log
I20260620 14:28:07.643360 13650 ts_tablet_manager.cc:1397] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260620 14:28:07.647395 13650 raft_consensus.cc:348] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d37af67f216464b9adfd8b014761a13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35435 } } peers { permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 } } peers { permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 } }
I20260620 14:28:07.647670 13650 raft_consensus.cc:374] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:07.647722 13650 raft_consensus.cc:729] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d37af67f216464b9adfd8b014761a13, State: Initialized, Role: FOLLOWER
I20260620 14:28:07.648178 13650 consensus_queue.cc:260] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d37af67f216464b9adfd8b014761a13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35435 } } peers { permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 } } peers { permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 } }
I20260620 14:28:07.648777 13344 heartbeater.cc:503] Master 127.0.98.62:42503 was elected leader, sending a full tablet report...
I20260620 14:28:07.649420 13650 ts_tablet_manager.cc:1428] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:28:07.651326 13651 tablet_bootstrap.cc:492] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: No bootstrap required, opened a new log
I20260620 14:28:07.651512 13651 ts_tablet_manager.cc:1397] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260620 14:28:07.655493 13651 raft_consensus.cc:348] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d37af67f216464b9adfd8b014761a13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35435 } } peers { permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 } } peers { permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 } }
I20260620 14:28:07.655738 13651 raft_consensus.cc:374] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:07.655797 13651 raft_consensus.cc:729] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52656129db14ff6a1d00209611b014c, State: Initialized, Role: FOLLOWER
I20260620 14:28:07.656241 13651 consensus_queue.cc:260] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d37af67f216464b9adfd8b014761a13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35435 } } peers { permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 } } peers { permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 } }
I20260620 14:28:07.657192 13651 ts_tablet_manager.cc:1428] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:28:07.657516 13612 heartbeater.cc:503] Master 127.0.98.62:42503 was elected leader, sending a full tablet report...
I20260620 14:28:07.714524 13654 raft_consensus.cc:482] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:07.714747 13654 raft_consensus.cc:504] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d37af67f216464b9adfd8b014761a13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35435 } } peers { permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 } } peers { permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 } }
I20260620 14:28:07.716157 13654 leader_election.cc:288] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d52656129db14ff6a1d00209611b014c (127.0.98.3:39247), ab0a0c44f5994460a1ab30e5b073ec7f (127.0.98.2:36509)
I20260620 14:28:07.721854 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.716898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47907 (local address 127.0.98.3:39247)
0620 14:28:07.717022 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:07.717026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:07.717041 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:28:07.717107 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:07.717111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:07.717160 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:07.717780 (+ 620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:07.718119 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.718921 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.720562 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.721135 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.721254 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.721315 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:07.721663 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 14:28:07.721732 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:28:07.722259 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.716508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39247 (local address 127.0.98.1:47907)
0620 14:28:07.716827 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:07.716843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:07.716948 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:07.717791 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:07.717795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:07.717816 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:07.717961 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:07.717971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.719067 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.719070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:07.720421 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:07.720431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.721411 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.721414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:07.721509 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.721533 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:28:07.722027 (+ 494us) client_negotiation.cc:240] Negotiation successful
0620 14:28:07.722102 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":26,"thread_start_us":117,"threads_started":1}
I20260620 14:28:07.722836 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.717592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33071 (local address 127.0.98.2:36509)
0620 14:28:07.717701 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:07.717705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:07.718154 (+ 449us) server_negotiation.cc:407] Connection header received
0620 14:28:07.718360 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:07.718364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:07.718417 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:07.718468 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:07.718893 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.719626 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.721128 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.721764 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.721847 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.721921 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:07.722646 (+ 725us) server_negotiation.cc:299] Negotiation successful
0620 14:28:07.722721 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:28:07.722823 13555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc97341efda14c7ebc7bd1332078d90f" candidate_uuid: "4d37af67f216464b9adfd8b014761a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d52656129db14ff6a1d00209611b014c" is_pre_election: true
I20260620 14:28:07.723153 13555 raft_consensus.cc:2454] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d37af67f216464b9adfd8b014761a13 in term 0.
I20260620 14:28:07.723308 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.717498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36509 (local address 127.0.98.1:33071)
0620 14:28:07.718040 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:07.718054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:07.718230 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:07.718605 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:07.718608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:07.718625 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:07.718747 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:07.718754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.719768 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.719771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:07.720984 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:07.720995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.722217 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.722220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:07.722489 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.722514 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:28:07.723059 (+ 545us) client_negotiation.cc:240] Negotiation successful
0620 14:28:07.723161 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":17,"thread_start_us":98,"threads_started":1}
I20260620 14:28:07.723655 13232 leader_election.cc:302] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d37af67f216464b9adfd8b014761a13, d52656129db14ff6a1d00209611b014c; no voters:
I20260620 14:28:07.723927 13654 raft_consensus.cc:2781] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:28:07.723888 13431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc97341efda14c7ebc7bd1332078d90f" candidate_uuid: "4d37af67f216464b9adfd8b014761a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" is_pre_election: true
I20260620 14:28:07.723999 13654 raft_consensus.cc:482] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:07.724040 13654 raft_consensus.cc:3037] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:07.724157 13431 raft_consensus.cc:2454] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d37af67f216464b9adfd8b014761a13 in term 0.
I20260620 14:28:07.725359 13654 raft_consensus.cc:504] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d37af67f216464b9adfd8b014761a13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35435 } } peers { permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 } } peers { permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 } }
I20260620 14:28:07.725708 13654 leader_election.cc:288] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [CANDIDATE]: Term 1 election: Requested vote from peers d52656129db14ff6a1d00209611b014c (127.0.98.3:39247), ab0a0c44f5994460a1ab30e5b073ec7f (127.0.98.2:36509)
I20260620 14:28:07.726166 13431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc97341efda14c7ebc7bd1332078d90f" candidate_uuid: "4d37af67f216464b9adfd8b014761a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f"
I20260620 14:28:07.726290 13431 raft_consensus.cc:3037] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:07.727522 13431 raft_consensus.cc:2454] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d37af67f216464b9adfd8b014761a13 in term 1.
I20260620 14:28:07.728698 13234 leader_election.cc:302] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d37af67f216464b9adfd8b014761a13, ab0a0c44f5994460a1ab30e5b073ec7f; no voters:
I20260620 14:28:07.729084 13654 raft_consensus.cc:2781] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:07.729122 13555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc97341efda14c7ebc7bd1332078d90f" candidate_uuid: "4d37af67f216464b9adfd8b014761a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d52656129db14ff6a1d00209611b014c"
I20260620 14:28:07.729256 13555 raft_consensus.cc:3037] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:07.729300 13654 raft_consensus.cc:686] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [term 1 LEADER]: Becoming Leader. State: Replica: 4d37af67f216464b9adfd8b014761a13, State: Running, Role: LEADER
I20260620 14:28:07.729638 13654 consensus_queue.cc:237] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d37af67f216464b9adfd8b014761a13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35435 } } peers { permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 } } peers { permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 } }
I20260620 14:28:07.730454 13555 raft_consensus.cc:2454] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d37af67f216464b9adfd8b014761a13 in term 1.
I20260620 14:28:07.732880 13146 catalog_manager.cc:5495] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d37af67f216464b9adfd8b014761a13 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4d37af67f216464b9adfd8b014761a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d37af67f216464b9adfd8b014761a13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 } health_report { overall_health: UNKNOWN } } }
W20260620 14:28:07.757272 13345 tablet.cc:2369] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:07.768193 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.763252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.0.98.1:35435)
0620 14:28:07.763464 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:07.763468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:07.763481 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:28:07.763554 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:07.763557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:07.763640 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:07.763749 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:07.764067 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.764798 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.766099 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:07.766437 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:07.766555 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:07.766889 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:07.767435 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:28:07.767551 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:07.767966 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 14:28:07.768022 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260620 14:28:07.774537 13431 raft_consensus.cc:1261] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f [term 1 FOLLOWER]: Refusing update from remote peer 4d37af67f216464b9adfd8b014761a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:07.774611 13555 raft_consensus.cc:1261] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c [term 1 FOLLOWER]: Refusing update from remote peer 4d37af67f216464b9adfd8b014761a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:07.775202 13654 consensus_queue.cc:1035] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0a0c44f5994460a1ab30e5b073ec7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:07.775401 13659 consensus_queue.cc:1035] T fc97341efda14c7ebc7bd1332078d90f P 4d37af67f216464b9adfd8b014761a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d52656129db14ff6a1d00209611b014c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:28:07.785638 13613 tablet.cc:2369] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:07.787380 13670 mvcc.cc:205] Tried to move back new op lower bound from 7298931457117093888 to 7298931456943353856. Current Snapshot: MvccSnapshot[applied={T|T < 7298931457117093888}]
I20260620 14:28:07.796386 13145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:07.796542 13145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:07.798959 13145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b393583dc3f4fd4994e67a3cdbcdf32: Sending DeleteTablet for 3 replicas of tablet fc97341efda14c7ebc7bd1332078d90f
I20260620 14:28:07.799611 13278 tablet_service.cc:1514] Processing DeleteTablet for tablet fc97341efda14c7ebc7bd1332078d90f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56828
I20260620 14:28:07.799772 13538 tablet_service.cc:1514] Processing DeleteTablet for tablet fc97341efda14c7ebc7bd1332078d90f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55194
I20260620 14:28:07.800019 13398 tablet_service.cc:1514] Processing DeleteTablet for tablet fc97341efda14c7ebc7bd1332078d90f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40584
I20260620 14:28:07.800467 13680 tablet_replica.cc:330] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: stopping tablet replica
I20260620 14:28:07.800809 13680 raft_consensus.cc:2229] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:07.800822 13679 tablet_replica.cc:330] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: stopping tablet replica
I20260620 14:28:07.801069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 13184
I20260620 14:28:07.801070 13679 raft_consensus.cc:2229] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:07.801229 13680 raft_consensus.cc:2258] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:07.801347 13679 raft_consensus.cc:2258] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:07.802891 13679 ts_tablet_manager.cc:1905] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:07.805550 13679 ts_tablet_manager.cc:1918] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:07.805639 13679 log.cc:1196] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/fc97341efda14c7ebc7bd1332078d90f
I20260620 14:28:07.805967 13679 ts_tablet_manager.cc:1939] T fc97341efda14c7ebc7bd1332078d90f P d52656129db14ff6a1d00209611b014c: Deleting consensus metadata
I20260620 14:28:07.806310 13680 ts_tablet_manager.cc:1905] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:07.807029 13131 catalog_manager.cc:4841] TS d52656129db14ff6a1d00209611b014c (127.0.98.3:39247): tablet fc97341efda14c7ebc7bd1332078d90f (table test-table [id=4f101d85747f42c1b61e12bbd9f22a7a]) successfully deleted
W20260620 14:28:07.809775 13130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35435 (error 108)
I20260620 14:28:07.809957 13680 ts_tablet_manager.cc:1918] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:07.810046 13680 log.cc:1196] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/fc97341efda14c7ebc7bd1332078d90f
I20260620 14:28:07.810348 13680 ts_tablet_manager.cc:1939] T fc97341efda14c7ebc7bd1332078d90f P ab0a0c44f5994460a1ab30e5b073ec7f: Deleting consensus metadata
I20260620 14:28:07.810523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 13348
I20260620 14:28:07.810817 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.810508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35435 (local address 127.0.0.1:56856)
0620 14:28:07.810667 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:07.810716 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 14:28:07.811303 13130 catalog_manager.cc:4584] TS 4d37af67f216464b9adfd8b014761a13 (127.0.98.1:35435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc97341efda14c7ebc7bd1332078d90f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35435: connect: Connection refused (error 111)
I20260620 14:28:07.817076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 13481
I20260620 14:28:07.817473 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:07.817233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36509 (local address 127.0.0.1:40596)
0620 14:28:07.817369 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:07.817416 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 14:28:07.822293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 13113
I20260620 14:28:07.828665 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:28:08.163901 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:28:08.163997 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:28:08.508375 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:28:08.508 UTC [12180] LOG: received smart shutdown request
2026-06-20 14:28:08.511 UTC [12180] LOG: background worker "logical replication launcher" (PID 12189) exited with exit code 1
2026-06-20 14:28:08.511 UTC [12184] LOG: shutting down
2026-06-20 14:28:08.635 UTC [12180] LOG: database system is shut down
2026-06-20T14:28:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45813 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:28:08 dist-test-slave-pk3j krb5kdc[13690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:28:08 dist-test-slave-pk3j krb5kdc[13690](info): set up 2 sockets
Jun 20 14:28:08 dist-test-slave-pk3j krb5kdc[13690](info): commencing operation
krb5kdc: starting...
W20260620 14:28:09.765115 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:28:09 dist-test-slave-pk3j krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:28:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:28:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:28:09.873049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40767
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40767,127.0.98.61:42429,127.0.98.60:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:09.974901 13703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:09.975116 13703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:09.975178 13703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:09.979055 13703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:09.979100 13703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:09.979118 13703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:09.979135 13703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:09.979184 13703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:09.983964 13703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40767,127.0.98.61:42429,127.0.98.60:43689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:09.984776 13703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:09.985692 13703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:09.993271 13709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:09.993271 13711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:09.993294 13708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:09.993425 13703 server_base.cc:1034] running on GCE node
I20260620 14:28:09.994022 13703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:09.994920 13703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:09.996107 13703 hybrid_clock.cc:648] HybridClock initialized: now 1781965689996103 us; error 32 us; skew 500 ppm
Jun 20 14:28:09 dist-test-slave-pk3j krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:09.998657 13703 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:09.999442 13703 webserver.cc:460] Webserver started at http://127.0.98.62:41335/ using document root <none> and password file <none>
I20260620 14:28:09.999920 13703 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:10.000005 13703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:10.000207 13703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:10.001988 13703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "10309d458077466a8365d2fffc49e5f4"
format_stamp: "Formatted at 2026-06-20 14:28:09 on dist-test-slave-pk3j"
server_key: "4d88e4cc711387d4ecfcad7a9e7a79b9"
server_key_iv: "58cffd4a197db50050ad771859b2bb84"
server_key_version: "encryptionkey@0"
I20260620 14:28:10.002535 13703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "10309d458077466a8365d2fffc49e5f4"
format_stamp: "Formatted at 2026-06-20 14:28:09 on dist-test-slave-pk3j"
server_key: "4d88e4cc711387d4ecfcad7a9e7a79b9"
server_key_iv: "58cffd4a197db50050ad771859b2bb84"
server_key_version: "encryptionkey@0"
I20260620 14:28:10.005713 13703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:28:10.007865 13718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:10.008944 13703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:10.009075 13703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "10309d458077466a8365d2fffc49e5f4"
format_stamp: "Formatted at 2026-06-20 14:28:09 on dist-test-slave-pk3j"
server_key: "4d88e4cc711387d4ecfcad7a9e7a79b9"
server_key_iv: "58cffd4a197db50050ad771859b2bb84"
server_key_version: "encryptionkey@0"
I20260620 14:28:10.009172 13703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:10.039165 13703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:10.041635 13703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:10.041800 13703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:10.048031 13703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40767
I20260620 14:28:10.048060 13770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40767 every 8 connection(s)
I20260620 14:28:10.048821 13703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:10.051294 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:10.054629 13771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:10.058046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 13703
I20260620 14:28:10.058161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:28:10.058495 392 external_mini_cluster.cc:1428] Setting key 67a2cee65b39adfec6d68750b4505393
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 14:28:10.070050 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.056719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40767 (local address 127.0.0.1:57396)
0620 14:28:10.057261 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.057304 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:10.058220 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:10.058899 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:10.058905 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:10.059374 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:10.059738 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.059758 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.061297 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.061300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.062511 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.062524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.063128 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.063134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.063287 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.063804 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:10.063829 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:10.065472 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:10.067372 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.067379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.067387 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.067587 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.067910 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.067914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.067916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.068017 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.068575 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:10.068580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:10.068689 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 14:28:10.068831 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 14:28:10.069173 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":128,"thread_start_us":136,"threads_started":1}
I20260620 14:28:10.070057 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.056719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.0.98.62:40767)
0620 14:28:10.057242 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.057259 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.057404 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:28:10.058507 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.058511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.058705 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.058841 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.059894 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.061158 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.062645 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.062973 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.063080 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.065617 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.065622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.065629 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.065647 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.067242 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.067689 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.067694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.067698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.067817 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.068106 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.068112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.068113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.068437 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.068576 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.068900 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.069098 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"spinlock_wait_cycles":45440,"thread_start_us":155,"threads_started":1}
I20260620 14:28:10.072314 13771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:10.072990 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.072725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42429 (local address 127.0.0.1:33448)
0620 14:28:10.072840 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.072920 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 14:28:10.073244 13720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42429: connect: Connection refused (error 111)
I20260620 14:28:10.074029 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.073844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42429 (local address 127.0.0.1:33460)
0620 14:28:10.073943 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.073983 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 14:28:10.074247 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42429: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42429: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:28:10.083472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42429
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40767,127.0.98.61:42429,127.0.98.60:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:10.123559 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } attempt: 1
I20260620 14:28:10.124197 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.123962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42429 (local address 127.0.0.1:33462)
0620 14:28:10.124088 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.124142 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:28:10.124869 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.124687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42429 (local address 127.0.0.1:33478)
0620 14:28:10.124779 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.124819 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:28:10.125000 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42429: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42429: connect: Connection refused (error 111)
W20260620 14:28:10.183943 13778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:10.184129 13778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:10.184180 13778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:10.188092 13778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:10.188138 13778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:10.188158 13778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:10.188174 13778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:10.188233 13778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:10.193230 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } attempt: 2
I20260620 14:28:10.193326 13778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40767,127.0.98.61:42429,127.0.98.60:43689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:10.194029 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.193713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42429 (local address 127.0.0.1:33482)
0620 14:28:10.193885 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.193951 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:28:10.194401 13778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:10.194713 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.194500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42429 (local address 127.0.0.1:33492)
0620 14:28:10.194619 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.194660 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:28:10.194869 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42429: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42429: connect: Connection refused (error 111)
I20260620 14:28:10.195372 13778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:10.202925 13784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:10.202926 13783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:10.203154 13778 server_base.cc:1034] running on GCE node
W20260620 14:28:10.203282 13786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:10.203774 13778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:10.204751 13778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:10.205922 13778 hybrid_clock.cc:648] HybridClock initialized: now 1781965690205897 us; error 37 us; skew 500 ppm
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:10.208407 13778 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:28:10.209142 13778 webserver.cc:460] Webserver started at http://127.0.98.61:32929/ using document root <none> and password file <none>
I20260620 14:28:10.209647 13778 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:10.209730 13778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:10.209934 13778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:10.211783 13778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/data/instance:
uuid: "0ec8fb28ad344095ae1b55b4337ff54b"
format_stamp: "Formatted at 2026-06-20 14:28:10 on dist-test-slave-pk3j"
server_key: "2a1b926049c63b187bf7569255cfebd4"
server_key_iv: "400d3dcd6ebcd01267e14c0e8ccae9f6"
server_key_version: "encryptionkey@0"
I20260620 14:28:10.212321 13778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/wal/instance:
uuid: "0ec8fb28ad344095ae1b55b4337ff54b"
format_stamp: "Formatted at 2026-06-20 14:28:10 on dist-test-slave-pk3j"
server_key: "2a1b926049c63b187bf7569255cfebd4"
server_key_iv: "400d3dcd6ebcd01267e14c0e8ccae9f6"
server_key_version: "encryptionkey@0"
I20260620 14:28:10.215739 13778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:28:10.217854 13793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:10.218909 13778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:10.219051 13778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/wal
uuid: "0ec8fb28ad344095ae1b55b4337ff54b"
format_stamp: "Formatted at 2026-06-20 14:28:10 on dist-test-slave-pk3j"
server_key: "2a1b926049c63b187bf7569255cfebd4"
server_key_iv: "400d3dcd6ebcd01267e14c0e8ccae9f6"
server_key_version: "encryptionkey@0"
I20260620 14:28:10.219149 13778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:10.233716 13778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:10.236140 13778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:10.236320 13778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:10.242386 13778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42429
I20260620 14:28:10.242451 13845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42429 every 8 connection(s)
I20260620 14:28:10.243225 13778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:28:10.245723 13846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:10.248489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 13778
I20260620 14:28:10.248610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-1/wal/instance
I20260620 14:28:10.248893 392 external_mini_cluster.cc:1428] Setting key 0031b84a63ec113251dd7cb87fe5c1fe
I20260620 14:28:10.249418 13846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:10.263777 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.251423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.0.98.62:40767)
0620 14:28:10.251582 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.251588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.252287 (+ 699us) server_negotiation.cc:407] Connection header received
0620 14:28:10.253245 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.253250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.253318 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.253437 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.254410 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.255230 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.256295 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.256698 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.256811 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.259954 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.259957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.259959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.259981 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.261634 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.262119 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.262126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.262130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.262222 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.262560 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.262563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.262565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.262795 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.262905 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.263397 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.263535 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:28:10.264252 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.251722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40767 (local address 127.0.0.1:57400)
0620 14:28:10.252127 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.252173 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:10.253021 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:10.253561 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:10.253567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:10.253870 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:10.254269 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.254281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.255366 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.255369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.256175 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.256183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.256839 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.256841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.256969 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.257537 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:10.257570 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:10.259784 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:10.261772 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.261776 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.261785 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.262017 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.262328 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.262330 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.262331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.262445 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.262928 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:10.262933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:10.263131 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 14:28:10.263299 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 14:28:10.263507 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":115,"threads_started":1}
I20260620 14:28:10.265516 13846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:28:10.273335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43689
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40767,127.0.98.61:42429,127.0.98.60:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:10.273965 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.265995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42429 (local address 127.0.0.1:33498)
0620 14:28:10.266129 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.266139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:10.266192 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:10.266678 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:10.266680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:10.266840 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:10.266916 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.266922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.267825 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.267831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.268381 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.268389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.269067 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.269071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.269162 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.269571 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:10.269581 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:10.270733 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:10.272453 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.272455 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.272456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.272672 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.272953 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.272956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.272958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.273050 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.273525 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:10.273528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:10.273604 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 14:28:10.273710 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:28:10.273823 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 14:28:10.274113 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.266102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.0.98.61:42429)
0620 14:28:10.266390 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.266397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.266412 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:28:10.266440 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.266444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.266564 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.266695 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.267020 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.267711 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.268491 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.268959 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.269072 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.270882 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.270890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.270898 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.270918 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.272344 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.272777 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.272789 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.272790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.272859 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.273143 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.273145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.273146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.273389 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.273525 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.273817 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.273923 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":124,"threads_started":1}
I20260620 14:28:10.275679 13846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:10.276351 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.276039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33690)
0620 14:28:10.276189 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.276270 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 14:28:10.276535 13797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
I20260620 14:28:10.277364 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.277075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33700)
0620 14:28:10.277225 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.277287 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:28:10.277518 13846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43689: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
I20260620 14:28:10.286026 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } attempt: 3
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:28:10.294286 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.286429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42429 (local address 127.0.0.1:33504)
0620 14:28:10.286538 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.286553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:10.286631 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:10.286892 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:10.286895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:10.287040 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:10.287458 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.287464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.288161 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.288166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.288700 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.288707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.289639 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.289643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.289744 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.290158 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:10.290169 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:10.291278 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:10.292857 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.292860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.292862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.293042 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.293408 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.293412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.293414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.293494 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.293828 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:10.293831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:10.293918 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 14:28:10.294013 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:28:10.294122 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:28:10.294294 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.286484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.0.98.61:42429)
0620 14:28:10.286596 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.286600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.286613 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:28:10.286742 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.286745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.286790 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.286871 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.287569 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.288059 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.288835 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.289521 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.289627 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.291464 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.291469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.291476 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.291496 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.292738 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.293147 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.293155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.293167 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.293294 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.293596 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.293599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.293601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.293740 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.293834 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.294067 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.294154 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:28:10.294909 13771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:10.295496 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.295312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33712)
0620 14:28:10.295405 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.295447 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:28:10.296062 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.295854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33724)
0620 14:28:10.295969 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.296011 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 14:28:10.296208 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43689: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
I20260620 14:28:10.326759 13846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } attempt: 1
I20260620 14:28:10.327355 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } attempt: 1
I20260620 14:28:10.327410 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.327145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33728)
0620 14:28:10.327276 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.327343 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:28:10.327893 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.327669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33736)
0620 14:28:10.327784 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.327833 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:28:10.328083 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.327850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33744)
0620 14:28:10.327949 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.327996 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:28:10.328275 13846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43689: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
I20260620 14:28:10.328572 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.328381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33756)
0620 14:28:10.328466 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.328517 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 14:28:10.328701 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43689: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
W20260620 14:28:10.377437 13852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:10.377640 13852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:10.377677 13852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:10.381572 13852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:10.381616 13852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:10.381635 13852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:10.381650 13852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:10.381686 13852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:10.386505 13852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40767,127.0.98.61:42429,127.0.98.60:43689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:10.387354 13852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:10.388170 13852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:10.395510 13858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:10.395524 13861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:10.395619 13852 server_base.cc:1034] running on GCE node
W20260620 14:28:10.395761 13859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:10.396324 13852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:10.396435 13846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } attempt: 2
I20260620 14:28:10.397100 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.396862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33770)
0620 14:28:10.396988 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.397035 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:28:10.397629 13852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:10.397802 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.397588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33780)
0620 14:28:10.397695 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.397744 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:28:10.397996 13846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43689: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
I20260620 14:28:10.398830 13852 hybrid_clock.cc:648] HybridClock initialized: now 1781965690398812 us; error 32 us; skew 500 ppm
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:10.401422 13852 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:28:10.402155 13852 webserver.cc:460] Webserver started at http://127.0.98.60:35067/ using document root <none> and password file <none>
I20260620 14:28:10.402634 13852 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:10.402719 13852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:10.402921 13852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:10.403905 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } attempt: 2
I20260620 14:28:10.404507 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.404285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33790)
0620 14:28:10.404405 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.404451 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:28:10.404989 13852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/data/instance:
uuid: "edd0a91c10f345d4ba650730bbc9e40a"
format_stamp: "Formatted at 2026-06-20 14:28:10 on dist-test-slave-pk3j"
server_key: "bb34cea548bbea2e30ef2385d6cb3ae3"
server_key_iv: "d15754e4ec10c562ff89ea6402c364f5"
server_key_version: "encryptionkey@0"
I20260620 14:28:10.405328 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.404995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33798)
0620 14:28:10.405145 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.405231 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 14:28:10.405447 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43689: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43689: connect: Connection refused (error 111)
I20260620 14:28:10.405609 13852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/wal/instance:
uuid: "edd0a91c10f345d4ba650730bbc9e40a"
format_stamp: "Formatted at 2026-06-20 14:28:10 on dist-test-slave-pk3j"
server_key: "bb34cea548bbea2e30ef2385d6cb3ae3"
server_key_iv: "d15754e4ec10c562ff89ea6402c364f5"
server_key_version: "encryptionkey@0"
I20260620 14:28:10.408876 13852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:10.411087 13868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:10.412196 13852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:10.412323 13852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/wal
uuid: "edd0a91c10f345d4ba650730bbc9e40a"
format_stamp: "Formatted at 2026-06-20 14:28:10 on dist-test-slave-pk3j"
server_key: "bb34cea548bbea2e30ef2385d6cb3ae3"
server_key_iv: "d15754e4ec10c562ff89ea6402c364f5"
server_key_version: "encryptionkey@0"
I20260620 14:28:10.412420 13852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:10.439826 13852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:10.442740 13852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:10.442924 13852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:10.449540 13852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43689
I20260620 14:28:10.449558 13920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43689 every 8 connection(s)
I20260620 14:28:10.450364 13852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:28:10.452875 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:10.456209 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:28:10.459415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 13852
I20260620 14:28:10.459548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781965536974027-392-0/minicluster-data/master-2/wal/instance
I20260620 14:28:10.459834 392 external_mini_cluster.cc:1428] Setting key 911ee48f6291c0041ac509affce110c9
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:28:10.469600 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.457936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.0.98.62:40767)
0620 14:28:10.458052 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.458056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.458770 (+ 714us) server_negotiation.cc:407] Connection header received
0620 14:28:10.459869 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.459890 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.459949 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.460012 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.461137 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.461699 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.462802 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.463032 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.463144 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.465713 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.465719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.465726 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.465757 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.467417 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.467984 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.467987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.467988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.468059 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.468413 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.468415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.468417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.468668 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.468773 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.469341 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.469428 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:28:10.470384 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.458250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40767 (local address 127.0.0.1:57404)
0620 14:28:10.458627 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.458664 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:10.459633 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:10.460196 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:10.460202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:10.460574 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:10.461006 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.461018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.461834 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.461836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.462678 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.462687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.463173 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.463177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.463335 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.463856 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:10.463871 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:10.465574 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:10.467523 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.467532 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.467544 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.467881 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.468149 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.468152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.468154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.468310 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.468781 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:10.468787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:10.469058 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 14:28:10.469275 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 14:28:10.469548 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":98,"threads_started":1}
I20260620 14:28:10.470391 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.460490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.0.98.60:43689)
0620 14:28:10.460814 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.460825 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.460858 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:28:10.460898 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.460902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.461038 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.461180 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.461881 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.462711 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.463494 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.463818 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.463935 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.466579 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.466585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.466604 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.466624 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.468314 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.468781 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.468786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.468788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.468866 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.469145 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.469148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.469150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.469412 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.469601 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.469917 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.470044 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":240,"thread_start_us":83,"threads_started":1}
I20260620 14:28:10.471741 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:10.479774 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.472206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42429 (local address 127.0.0.1:33512)
0620 14:28:10.472333 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.472346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:10.472411 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:10.472651 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:10.472654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:10.472813 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:10.472946 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.472953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.473822 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.473825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.474413 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.474423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.474816 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.474820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.474927 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.475346 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:10.475359 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:10.476572 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:10.478434 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.478436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.478438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.478613 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.478896 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.478900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.478902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.478990 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.479342 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:10.479346 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:10.479431 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 14:28:10.479531 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:28:10.479640 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 14:28:10.479816 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.472292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.0.98.61:42429)
0620 14:28:10.472410 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.472415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.472428 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:28:10.472497 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.472499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.472543 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.472635 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.473068 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.473710 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.474516 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.474696 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.474783 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.476710 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.476713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.476715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.476735 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.478329 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.478723 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.478727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.478729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.478805 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.479078 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.479083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.479084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.479241 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.479340 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.479592 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.479686 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:28:10.480270 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.472540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.0.98.62:40767)
0620 14:28:10.472648 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.472651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.472662 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:28:10.472747 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.472754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.472784 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.472840 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.473491 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.474024 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.474889 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.475124 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.475229 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.477471 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.477474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.477476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.477498 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.478721 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.479121 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.479124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.479125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.479202 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.479460 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.479465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.479466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.479620 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.479816 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.479943 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.480043 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:28:10.480772 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:28:10.488938 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.481201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33820)
0620 14:28:10.481389 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.481401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:10.481488 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:10.481731 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:10.481734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:10.481863 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:10.481921 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.481926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.482762 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.482766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.483297 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.483305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.483900 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.483903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.483986 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.484514 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:10.484526 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:10.485608 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:10.487499 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.487502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.487504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.487729 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.488003 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.488006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.488008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.488093 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.488478 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:10.488482 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:10.488566 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 14:28:10.488667 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:28:10.488783 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260620 14:28:10.489095 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.481331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.0.98.61:42429)
0620 14:28:10.481427 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.481431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.481444 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:28:10.481545 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.481548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.481587 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.481651 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.482278 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.482819 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.483527 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.483708 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.483922 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.486229 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.486232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.486235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.486257 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.487532 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.487929 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.487931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.487932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.488037 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.488318 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.488320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.488321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.488501 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.488710 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.488829 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.488921 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:28:10.489131 13846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } attempt: 3
I20260620 14:28:10.489053 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.481271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.0.98.60:43689)
0620 14:28:10.481366 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.481369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.481447 (+ 78us) server_negotiation.cc:407] Connection header received
0620 14:28:10.481569 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.481572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.481612 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.481699 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.482028 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.482656 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.483408 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.483692 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.483762 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.485732 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.485735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.485738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.485758 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.487359 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.487816 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.487819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.487821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.487912 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.488179 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.488185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.488188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.488374 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.488450 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.488726 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.488852 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:28:10.492605 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:28:10.492735 13921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a: Bootstrap starting.
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:28:10.495175 13921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:10.496109 13921 log.cc:824] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:10.498821 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.489606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33834)
0620 14:28:10.489764 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.489777 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:10.489848 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:10.490168 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:10.490171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:10.490335 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:10.490425 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.490432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.491605 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.491610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.492256 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.492265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.492829 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.492838 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.492916 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.493381 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:10.493394 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:10.494785 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:10.497281 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.497284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.497286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.497543 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.497861 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.497864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.497866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.497960 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.498394 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:10.498397 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:10.498483 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 14:28:10.498567 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:28:10.498670 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 14:28:10.500001 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.489639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.0.98.60:43689)
0620 14:28:10.489947 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.489959 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.489974 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:28:10.490010 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.490014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.490049 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.490140 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.490510 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.491208 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.492381 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.492717 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.492825 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.494951 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.494954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.494957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.494978 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.497125 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.497658 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.497662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.497666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.497741 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.498060 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.498063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.498065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.498270 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.498354 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.499678 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.499843 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":95,"threads_started":1}
I20260620 14:28:10.500998 13921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a: No bootstrap required, opened a new log
I20260620 14:28:10.503859 13846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b: Bootstrap starting.
I20260620 14:28:10.504171 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.495429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.0.98.62:40767)
0620 14:28:10.495552 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.495556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.495569 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:28:10.495646 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.495651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.495691 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.495757 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.496671 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.497445 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.498524 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.498816 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.498910 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.500410 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.500415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.500418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.500444 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.502207 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.502575 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.502580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.502583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.502664 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.502991 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.502994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.502997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.503195 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.503322 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.503919 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.504025 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:28:10.504595 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } attempt: 3
I20260620 14:28:10.504860 13921 raft_consensus.cc:348] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } }
I20260620 14:28:10.505096 13921 raft_consensus.cc:374] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:10.505100 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.495599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.0.98.60:43689)
0620 14:28:10.495678 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.495682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.495820 (+ 138us) server_negotiation.cc:407] Connection header received
0620 14:28:10.495976 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.495980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.496019 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.496072 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.496941 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.497640 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.498650 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.498958 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.499023 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.501381 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.501386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.501392 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.501416 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.503214 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.503658 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.503661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.503663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.503741 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.504048 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.504051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.504053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.504229 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.504323 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.504859 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.504969 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:28:10.505159 13921 raft_consensus.cc:729] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd0a91c10f345d4ba650730bbc9e40a, State: Initialized, Role: FOLLOWER
I20260620 14:28:10.506004 13921 consensus_queue.cc:260] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } }
I20260620 14:28:10.506438 13846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:10.507275 13846 log.cc:824] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:10.507378 13921 sys_catalog.cc:564] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:10.507799 13935 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } } }
I20260620 14:28:10.507987 13935 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:10.509359 13846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b: No bootstrap required, opened a new log
Jun 20 14:28:10 dist-test-slave-pk3j krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:28:10.513715 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.505056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43689 (local address 127.0.0.1:33848)
0620 14:28:10.505266 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:10.505278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:10.505340 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:10.505855 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:10.505858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:10.506016 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:10.506084 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.506089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.506952 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.506957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.507534 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:10.507543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.508167 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.508174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:10.508254 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.508670 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:10.508687 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:10.510159 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:10.512103 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.512106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.512108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.512358 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.512710 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:10.512713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:10.512715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:10.512797 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:10.513271 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:10.513274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:10.513361 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 14:28:10.513459 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:28:10.513576 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260620 14:28:10.513994 13846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } }
I20260620 14:28:10.514248 13846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:10.514339 13846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec8fb28ad344095ae1b55b4337ff54b, State: Initialized, Role: FOLLOWER
I20260620 14:28:10.514437 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.505520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.0.98.60:43689)
0620 14:28:10.505608 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.505611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.505622 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:28:10.505650 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.505652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.505684 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.505756 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.506212 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.506830 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.507648 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.507950 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.508060 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.510328 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.510332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.510334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.510356 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.511967 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.512511 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.512514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.512516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.512590 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.512922 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.512925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.512927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.513106 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.513194 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.514176 (+ 982us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.514292 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:28:10.514845 13846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } }
I20260620 14:28:10.510790 13852 master_runner.cc:186] Error getting master registration for 127.0.98.62:40767: OK, instance_id { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" instance_seqno: 1781965690046161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:28:10.515363 13852 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:28:10.515725 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:10.495129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.0.98.61:42429)
0620 14:28:10.495220 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:10.495226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:10.496868 (+ 1642us) server_negotiation.cc:407] Connection header received
0620 14:28:10.496907 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:10.496910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:10.496966 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:10.497037 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:10.498029 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.498655 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.509354 (+ 10699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:10.509731 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:10.509842 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:10.512114 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:10.512117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:10.512120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:10.512142 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:10.513817 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.514275 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.514278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.514279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.514388 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:10.514684 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:10.514687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:10.514689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:10.514864 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:10.514990 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:10.515544 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 14:28:10.515637 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:28:10.516258 13846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:10.517375 13771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4: Bootstrap starting.
I20260620 14:28:10.517433 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } } }
I20260620 14:28:10.517576 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:10.519850 13771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4: Neither blocks nor log segments found. Creating new log.
W20260620 14:28:10.520531 13947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:28:10.520534 13959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:28:10.520596 13959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 14:28:10.520664 13947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:28:10.521015 13771 log.cc:824] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:10.522876 13771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4: No bootstrap required, opened a new log
I20260620 14:28:10.519335 13778 master_runner.cc:186] Error getting master registration for 127.0.98.62:40767: OK, instance_id { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" instance_seqno: 1781965690046161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:28:10.523100 13778 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:28:10.526554 13771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } }
I20260620 14:28:10.526754 13771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:10.526819 13771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10309d458077466a8365d2fffc49e5f4, State: Initialized, Role: FOLLOWER
I20260620 14:28:10.527216 13771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } }
I20260620 14:28:10.527722 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } } }
I20260620 14:28:10.527853 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:10.528398 13771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:28:10.533100 13974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:28:10.533174 13974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:28:10.536396 13703 master_runner.cc:263] No leader master found from master 10309d458077466a8365d2fffc49e5f4
I20260620 14:28:10.536465 13703 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:28:10.543116 13724 tablet.cc:2369] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:28:10.694152 13874 tablet.cc:2369] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:28:10.737669 13799 tablet.cc:2369] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:10.780979 13963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:10.781128 13963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } }
I20260620 14:28:10.781965 13963 leader_election.cc:288] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec8fb28ad344095ae1b55b4337ff54b (127.0.98.61:42429), edd0a91c10f345d4ba650730bbc9e40a (127.0.98.60:43689)
I20260620 14:28:10.782366 13895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10309d458077466a8365d2fffc49e5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd0a91c10f345d4ba650730bbc9e40a" is_pre_election: true
I20260620 14:28:10.782362 13820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10309d458077466a8365d2fffc49e5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" is_pre_election: true
I20260620 14:28:10.782636 13895 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10309d458077466a8365d2fffc49e5f4 in term 0.
I20260620 14:28:10.782635 13820 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10309d458077466a8365d2fffc49e5f4 in term 0.
I20260620 14:28:10.783073 13722 leader_election.cc:302] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10309d458077466a8365d2fffc49e5f4, edd0a91c10f345d4ba650730bbc9e40a; no voters:
I20260620 14:28:10.783383 13963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:28:10.783473 13963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:10.783519 13963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:10.784482 13963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } }
I20260620 14:28:10.784759 13963 leader_election.cc:288] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec8fb28ad344095ae1b55b4337ff54b (127.0.98.61:42429), edd0a91c10f345d4ba650730bbc9e40a (127.0.98.60:43689)
I20260620 14:28:10.785012 13820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10309d458077466a8365d2fffc49e5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec8fb28ad344095ae1b55b4337ff54b"
I20260620 14:28:10.785043 13895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10309d458077466a8365d2fffc49e5f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd0a91c10f345d4ba650730bbc9e40a"
I20260620 14:28:10.785138 13820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:10.785173 13895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:10.786079 13820 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10309d458077466a8365d2fffc49e5f4 in term 1.
I20260620 14:28:10.786098 13895 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10309d458077466a8365d2fffc49e5f4 in term 1.
I20260620 14:28:10.786386 13720 leader_election.cc:302] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec8fb28ad344095ae1b55b4337ff54b, 10309d458077466a8365d2fffc49e5f4; no voters:
I20260620 14:28:10.786571 13963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:10.786768 13963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [term 1 LEADER]: Becoming Leader. State: Replica: 10309d458077466a8365d2fffc49e5f4, State: Running, Role: LEADER
I20260620 14:28:10.787079 13963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } }
I20260620 14:28:10.788841 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10309d458077466a8365d2fffc49e5f4. Latest consensus state: current_term: 1 leader_uuid: "10309d458077466a8365d2fffc49e5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } } }
I20260620 14:28:10.788955 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:10.789348 13978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:10.790125 13978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:10.794631 13895 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [term 1 FOLLOWER]: Refusing update from remote peer 10309d458077466a8365d2fffc49e5f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:10.794664 13820 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [term 1 FOLLOWER]: Refusing update from remote peer 10309d458077466a8365d2fffc49e5f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:10.795193 13963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:10.795372 13976 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:10.798712 13935 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10309d458077466a8365d2fffc49e5f4. Latest consensus state: current_term: 1 leader_uuid: "10309d458077466a8365d2fffc49e5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } } }
I20260620 14:28:10.798727 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10309d458077466a8365d2fffc49e5f4. Latest consensus state: current_term: 1 leader_uuid: "10309d458077466a8365d2fffc49e5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } } }
I20260620 14:28:10.798811 13935 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:10.798822 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:10.801164 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10309d458077466a8365d2fffc49e5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } } }
I20260620 14:28:10.801192 13984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10309d458077466a8365d2fffc49e5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } } }
I20260620 14:28:10.801301 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:10.801329 13984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:10.801378 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10309d458077466a8365d2fffc49e5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } } }
I20260620 14:28:10.801476 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:10.801496 13935 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10309d458077466a8365d2fffc49e5f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10309d458077466a8365d2fffc49e5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40767 } } peers { permanent_uuid: "0ec8fb28ad344095ae1b55b4337ff54b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42429 } } peers { permanent_uuid: "edd0a91c10f345d4ba650730bbc9e40a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43689 } } }
I20260620 14:28:10.801599 13935 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:28:10.801733 13981 mvcc.cc:205] Tried to move back new op lower bound from 7298931469488697344 to 7298931469466284032. Current Snapshot: MvccSnapshot[applied={T|T < 7298931469488697344}]
I20260620 14:28:10.801788 13980 mvcc.cc:205] Tried to move back new op lower bound from 7298931469488697344 to 7298931469466284032. Current Snapshot: MvccSnapshot[applied={T|T < 7298931469488697344}]
I20260620 14:28:10.802593 13978 catalog_manager.cc:1347] Generated new cluster ID: 76581727f28648e8b8f9bb7073343773
I20260620 14:28:10.802646 13978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:10.818521 13978 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:28:10.819590 13978 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:10.825990 13978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10309d458077466a8365d2fffc49e5f4: Generated new TSK 0
I20260620 14:28:10.826738 13978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:28:11.522058 13947 catalog_manager.cc:1259] Loaded cluster ID: 76581727f28648e8b8f9bb7073343773
I20260620 14:28:11.522058 13959 catalog_manager.cc:1259] Loaded cluster ID: 76581727f28648e8b8f9bb7073343773
I20260620 14:28:11.522142 13947 catalog_manager.cc:1552] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a: loading cluster ID for follower catalog manager: success
I20260620 14:28:11.522142 13959 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b: loading cluster ID for follower catalog manager: success
I20260620 14:28:11.524271 13947 catalog_manager.cc:1574] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a: acquiring CA information for follower catalog manager: success
I20260620 14:28:11.524271 13959 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b: acquiring CA information for follower catalog manager: success
I20260620 14:28:11.525666 13959 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ec8fb28ad344095ae1b55b4337ff54b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:28:11.525666 13947 catalog_manager.cc:1602] T 00000000000000000000000000000000 P edd0a91c10f345d4ba650730bbc9e40a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:28:11.883340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 13703
I20260620 14:28:11.889379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 13778
I20260620 14:28:11.894656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 13852
2026-06-20T14:28:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3147 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:28:11 dist-test-slave-pk3j krb5kdc[13993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:28:11 dist-test-slave-pk3j krb5kdc[13993](info): set up 2 sockets
Jun 20 14:28:11 dist-test-slave-pk3j krb5kdc[13993](info): commencing operation
krb5kdc: starting...
W20260620 14:28:12.917328 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:28:12 dist-test-slave-pk3j krb5kdc[13993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:28:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:28:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:28:13.023540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41471
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:13.129694 14006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:13.129912 14006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:13.129949 14006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:13.133872 14006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:13.133913 14006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:13.133931 14006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:13.133946 14006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:13.133983 14006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:13.138924 14006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:13.139789 14006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:13.140692 14006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:13.148255 14011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:13.148272 14012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:13.148352 14014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:13.148615 14006 server_base.cc:1034] running on GCE node
I20260620 14:28:13.149106 14006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:13.150113 14006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:13.151268 14006 hybrid_clock.cc:648] HybridClock initialized: now 1781965693151245 us; error 36 us; skew 500 ppm
Jun 20 14:28:13 dist-test-slave-pk3j krb5kdc[13993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:13.154023 14006 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:13.154858 14006 webserver.cc:460] Webserver started at http://127.0.98.62:35157/ using document root <none> and password file <none>
I20260620 14:28:13.155375 14006 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:13.155432 14006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:13.155673 14006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:13.157629 14006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "ed8648fb7e684c309b9089b8fcdec663"
format_stamp: "Formatted at 2026-06-20 14:28:13 on dist-test-slave-pk3j"
server_key: "663c8f8937909a3b874a431e48e57b69"
server_key_iv: "bbb910a2eae6d2b1ede0e7127294af49"
server_key_version: "encryptionkey@0"
I20260620 14:28:13.158166 14006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed8648fb7e684c309b9089b8fcdec663"
format_stamp: "Formatted at 2026-06-20 14:28:13 on dist-test-slave-pk3j"
server_key: "663c8f8937909a3b874a431e48e57b69"
server_key_iv: "bbb910a2eae6d2b1ede0e7127294af49"
server_key_version: "encryptionkey@0"
I20260620 14:28:13.161594 14006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:28:13.163807 14021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:13.164870 14006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:13.164984 14006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "ed8648fb7e684c309b9089b8fcdec663"
format_stamp: "Formatted at 2026-06-20 14:28:13 on dist-test-slave-pk3j"
server_key: "663c8f8937909a3b874a431e48e57b69"
server_key_iv: "bbb910a2eae6d2b1ede0e7127294af49"
server_key_version: "encryptionkey@0"
I20260620 14:28:13.165078 14006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:13.183866 14006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:13.186802 14006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:13.186996 14006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:13.193722 14006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41471
I20260620 14:28:13.193737 14073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41471 every 8 connection(s)
I20260620 14:28:13.194634 14006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:13.197089 14074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:13.198406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 14006
I20260620 14:28:13.198535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:28:13.198805 392 external_mini_cluster.cc:1428] Setting key 4c16a5a31dbab011ad60693462cf5143
I20260620 14:28:13.202766 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663: Bootstrap starting.
I20260620 14:28:13.205134 14074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:13.206004 14074 log.cc:824] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:28:13 dist-test-slave-pk3j krb5kdc[13993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:13.207930 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663: No bootstrap required, opened a new log
I20260620 14:28:13.211709 14074 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8648fb7e684c309b9089b8fcdec663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260620 14:28:13.211939 14074 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:13.212014 14074 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed8648fb7e684c309b9089b8fcdec663, State: Initialized, Role: FOLLOWER
I20260620 14:28:13.212109 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:13.199918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.0.98.62:41471)
0620 14:28:13.200277 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:13.200299 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:13.200336 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:28:13.200911 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:13.200926 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:13.201200 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:13.201506 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:13.202396 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:13.203346 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:13.204530 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:13.204950 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:13.205057 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:13.207624 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:13.207630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:13.207643 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:13.207670 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:13.209406 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:13.209945 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:13.209949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:13.209953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:13.210052 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:13.210330 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:13.210332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:13.210333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:13.210653 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:13.210831 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:13.211104 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 14:28:13.211373 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":115,"threads_started":1}
I20260620 14:28:13.212491 14074 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8648fb7e684c309b9089b8fcdec663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260620 14:28:13.212642 14074 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:13.212704 14074 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:13.212790 14074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:13.213683 14074 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8648fb7e684c309b9089b8fcdec663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260620 14:28:13.214039 14074 leader_election.cc:302] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed8648fb7e684c309b9089b8fcdec663; no voters:
I20260620 14:28:13.214355 14074 leader_election.cc:288] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:28:13.214448 14079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:13.214675 14079 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [term 1 LEADER]: Becoming Leader. State: Replica: ed8648fb7e684c309b9089b8fcdec663, State: Running, Role: LEADER
I20260620 14:28:13.214994 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:28:13.214984 14079 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8648fb7e684c309b9089b8fcdec663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } }
I20260620 14:28:13.215523 14074 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:13.216320 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed8648fb7e684c309b9089b8fcdec663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8648fb7e684c309b9089b8fcdec663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } } }
I20260620 14:28:13.216449 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:13.216665 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed8648fb7e684c309b9089b8fcdec663. Latest consensus state: current_term: 1 leader_uuid: "ed8648fb7e684c309b9089b8fcdec663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed8648fb7e684c309b9089b8fcdec663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41471 } } }
I20260620 14:28:13.216737 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:13.217041 14088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:13.219894 14088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:13.224419 14088 catalog_manager.cc:1347] Generated new cluster ID: 2d6d5931fb97495ead40210eeff5a7bb
I20260620 14:28:13.224476 14088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:13.239928 14088 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:28:13.240717 14088 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:13.249184 14088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed8648fb7e684c309b9089b8fcdec663: Generated new TSK 0
I20260620 14:28:13.249756 14088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:28:13.270617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 14006
2026-06-20T14:28:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1373 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:28:13 dist-test-slave-pk3j krb5kdc[14101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:28:13 dist-test-slave-pk3j krb5kdc[14101](info): set up 2 sockets
Jun 20 14:28:13 dist-test-slave-pk3j krb5kdc[14101](info): commencing operation
krb5kdc: starting...
W20260620 14:28:14.281245 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:28:14 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:28:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:28:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:28:14.391899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45975
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:14.497897 14114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:14.498106 14114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:14.498169 14114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:14.501948 14114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:14.501993 14114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:14.502012 14114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:14.502055 14114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:14.502089 14114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:14.507158 14114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:14.508054 14114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:14.508960 14114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:14.516528 14122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:14.516549 14120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:14.516594 14119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:14.516722 14114 server_base.cc:1034] running on GCE node
I20260620 14:28:14.517403 14114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:14.518412 14114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:14.519596 14114 hybrid_clock.cc:648] HybridClock initialized: now 1781965694519577 us; error 44 us; skew 500 ppm
Jun 20 14:28:14 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:14.527159 14114 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:14.528007 14114 webserver.cc:460] Webserver started at http://127.0.98.62:38323/ using document root <none> and password file <none>
I20260620 14:28:14.528514 14114 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:14.528601 14114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:14.528812 14114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:14.530740 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "db791c82f21d40f09440c03474b98e71"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "5d63bcecf87e82d5bee0ba10a6686909"
server_key_iv: "35d7a115ec66065da7efaf542d1e90c6"
server_key_version: "encryptionkey@0"
I20260620 14:28:14.531297 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "db791c82f21d40f09440c03474b98e71"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "5d63bcecf87e82d5bee0ba10a6686909"
server_key_iv: "35d7a115ec66065da7efaf542d1e90c6"
server_key_version: "encryptionkey@0"
I20260620 14:28:14.534744 14114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:28:14.536851 14129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:14.538019 14114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:14.538159 14114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "db791c82f21d40f09440c03474b98e71"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "5d63bcecf87e82d5bee0ba10a6686909"
server_key_iv: "35d7a115ec66065da7efaf542d1e90c6"
server_key_version: "encryptionkey@0"
I20260620 14:28:14.538259 14114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:14.552932 14114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:14.555732 14114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:14.555917 14114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:14.562723 14114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45975
I20260620 14:28:14.562999 14181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45975 every 8 connection(s)
I20260620 14:28:14.563601 14114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:14.566212 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:14.566911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 14114
I20260620 14:28:14.567024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:28:14.567313 392 external_mini_cluster.cc:1428] Setting key 774996c6d254a8ff94ca903a8c424323
I20260620 14:28:14.571727 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Bootstrap starting.
I20260620 14:28:14.573752 14182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:14.574389 14182 log.cc:824] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:28:14 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:14.576234 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: No bootstrap required, opened a new log
I20260620 14:28:14.580674 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:14.568594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38930 (local address 127.0.98.62:45975)
0620 14:28:14.569016 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:14.569028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:14.569058 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:28:14.569681 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:14.569697 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:14.570006 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:14.570322 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:14.571104 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:14.572073 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:14.572886 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:14.573410 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:14.573573 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:14.576015 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:14.576022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:14.576035 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:14.576062 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:14.577898 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:14.578291 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:14.578299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:14.578306 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:14.578453 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:14.578700 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:14.578705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:14.578707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:14.579163 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:14.579320 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:14.579565 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:28:14.579820 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260620 14:28:14.580653 14182 raft_consensus.cc:348] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:14.580905 14182 raft_consensus.cc:374] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:14.580977 14182 raft_consensus.cc:729] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db791c82f21d40f09440c03474b98e71, State: Initialized, Role: FOLLOWER
I20260620 14:28:14.581434 14182 consensus_queue.cc:260] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:14.581588 14182 raft_consensus.cc:388] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:14.581701 14182 raft_consensus.cc:482] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:14.581809 14182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:14.582631 14182 raft_consensus.cc:504] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:14.582930 14182 leader_election.cc:302] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db791c82f21d40f09440c03474b98e71; no voters:
I20260620 14:28:14.583225 14182 leader_election.cc:288] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:28:14.583352 14187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:14.583561 14187 raft_consensus.cc:686] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 1 LEADER]: Becoming Leader. State: Replica: db791c82f21d40f09440c03474b98e71, State: Running, Role: LEADER
I20260620 14:28:14.583878 14187 consensus_queue.cc:237] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:14.584286 14182 sys_catalog.cc:564] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:14.585448 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db791c82f21d40f09440c03474b98e71. Latest consensus state: current_term: 1 leader_uuid: "db791c82f21d40f09440c03474b98e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } } }
I20260620 14:28:14.585455 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db791c82f21d40f09440c03474b98e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } } }
I20260620 14:28:14.585620 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:14.585635 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:14.585980 14196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:14.588765 14196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:14.593694 14196 catalog_manager.cc:1347] Generated new cluster ID: 8ea5eda6f33840d8a02dc0e9e52a86d4
I20260620 14:28:14.593750 14196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:14.612490 14196 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:28:14.613317 14196 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:14.618214 14196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Generated new TSK 0
I20260620 14:28:14.618939 14196 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:28:14.658388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45975
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:14.782002 14208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:14.782220 14208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:14.782258 14208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:14.786362 14208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:14.786410 14208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:14.786510 14208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:14.791481 14208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:14.792309 14208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:14.793175 14208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:14.801963 14213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:14.801982 14214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:14.802011 14216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:14.802402 14208 server_base.cc:1034] running on GCE node
I20260620 14:28:14.802773 14208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:14.803364 14208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:14.804539 14208 hybrid_clock.cc:648] HybridClock initialized: now 1781965694804519 us; error 30 us; skew 500 ppm
Jun 20 14:28:14 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:14.807425 14208 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:14.808224 14208 webserver.cc:460] Webserver started at http://127.0.98.1:38687/ using document root <none> and password file <none>
I20260620 14:28:14.808686 14208 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:14.808741 14208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:14.808884 14208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:14.810683 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "003b0d6e179a459e8dbd975710b76112"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "7f60dfb9aada4015f8c6a56a36cd8f65"
server_key_iv: "a4a5aecaf46c223b85ac865ce74239b4"
server_key_version: "encryptionkey@0"
I20260620 14:28:14.811189 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "003b0d6e179a459e8dbd975710b76112"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "7f60dfb9aada4015f8c6a56a36cd8f65"
server_key_iv: "a4a5aecaf46c223b85ac865ce74239b4"
server_key_version: "encryptionkey@0"
I20260620 14:28:14.814504 14208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:28:14.816648 14223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:14.817845 14208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:14.817972 14208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "003b0d6e179a459e8dbd975710b76112"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "7f60dfb9aada4015f8c6a56a36cd8f65"
server_key_iv: "a4a5aecaf46c223b85ac865ce74239b4"
server_key_version: "encryptionkey@0"
I20260620 14:28:14.818073 14208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:14.830173 14208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:14.832582 14208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:14.832763 14208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:14.833320 14208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:14.834036 14208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:14.834105 14208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:14.834168 14208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:14.834218 14208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:14.842419 14208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44607
I20260620 14:28:14.842694 14336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44607 every 8 connection(s)
I20260620 14:28:14.843267 14208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:14.843878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 14208
I20260620 14:28:14.844048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:28:14.844341 392 external_mini_cluster.cc:1428] Setting key 554af59380f06a3fd2ec8f401ce7a54f
Jun 20 14:28:14 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:28:14.855702 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:14.845068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38127 (local address 127.0.98.62:45975)
0620 14:28:14.845196 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:14.845241 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:14.845981 (+ 740us) server_negotiation.cc:407] Connection header received
0620 14:28:14.846797 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:14.846800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:14.846852 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:14.846928 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:14.847951 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:14.848467 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:14.849588 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:14.849857 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:14.849970 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:14.852239 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:14.852242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:14.852245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:14.852270 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:14.853718 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:14.854207 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:14.854212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:14.854214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:14.854294 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:14.854629 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:14.854632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:14.854634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:14.854810 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:14.854915 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:14.855416 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 14:28:14.855537 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:28:14.856380 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:14.845355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:38127)
0620 14:28:14.845842 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:14.845879 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:14.846601 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:14.847074 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:14.847080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:14.847393 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:14.847794 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:14.847806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:14.848588 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:14.848592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:14.849467 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:14.849476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:14.850003 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:14.850007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:14.850168 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:14.850742 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:14.850757 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:14.852080 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:14.853844 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:14.853850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:14.853862 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:14.854101 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:14.854393 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:14.854396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:14.854398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:14.854528 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:14.854914 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:14.854920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:14.855188 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 14:28:14.855372 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 14:28:14.855649 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":134,"threads_started":1}
I20260620 14:28:14.857575 14337 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45975
I20260620 14:28:14.857801 14337 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:14.858316 14337 heartbeater.cc:511] Master 127.0.98.62:45975 requested a full tablet report, sending...
I20260620 14:28:14.860000 14146 ts_manager.cc:194] Registered new tserver with Master: 003b0d6e179a459e8dbd975710b76112 (127.0.98.1:44607)
I20260620 14:28:14.861601 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:28:14.869716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45975
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:14.971662 14342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:14.971859 14342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:14.971897 14342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:14.975548 14342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:14.975587 14342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:14.975634 14342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:14.980243 14342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:14.980984 14342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:14.981763 14342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:14.990522 14348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:14.990504 14350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:14.990628 14342 server_base.cc:1034] running on GCE node
W20260620 14:28:14.990537 14347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:14.991168 14342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:14.991794 14342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:14.992971 14342 hybrid_clock.cc:648] HybridClock initialized: now 1781965694992967 us; error 29 us; skew 500 ppm
Jun 20 14:28:14 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:14.995543 14342 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:14.996270 14342 webserver.cc:460] Webserver started at http://127.0.98.2:36607/ using document root <none> and password file <none>
I20260620 14:28:14.996771 14342 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:14.996858 14342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:14.997066 14342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:14.999053 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "5fbc961a9ac2442fa66df3a23bd39ae3"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "5da288b13e09210e304fbc2e09ddd732"
server_key_iv: "c0312be85d13f6f04bc65fe687fd6013"
server_key_version: "encryptionkey@0"
I20260620 14:28:14.999562 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5fbc961a9ac2442fa66df3a23bd39ae3"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "5da288b13e09210e304fbc2e09ddd732"
server_key_iv: "c0312be85d13f6f04bc65fe687fd6013"
server_key_version: "encryptionkey@0"
I20260620 14:28:15.002766 14342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 14:28:15.004889 14357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:15.006034 14342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:28:15.006166 14342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "5fbc961a9ac2442fa66df3a23bd39ae3"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "5da288b13e09210e304fbc2e09ddd732"
server_key_iv: "c0312be85d13f6f04bc65fe687fd6013"
server_key_version: "encryptionkey@0"
I20260620 14:28:15.006268 14342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:15.019910 14342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:15.022888 14342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:15.023062 14342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:15.023554 14342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:15.024269 14342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:15.024339 14342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:15.024403 14342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:15.024451 14342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:15.032537 14342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43667
I20260620 14:28:15.032595 14470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43667 every 8 connection(s)
I20260620 14:28:15.033385 14342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:15.034598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 14342
I20260620 14:28:15.034734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:28:15.035013 392 external_mini_cluster.cc:1428] Setting key 7788a29b14230b241a65960423f7fd18
Jun 20 14:28:15 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:28:15.046835 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.035114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43211 (local address 127.0.98.62:45975)
0620 14:28:15.035241 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:15.035246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:15.036144 (+ 898us) server_negotiation.cc:407] Connection header received
0620 14:28:15.037530 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:15.037533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:15.037586 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:15.037661 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:15.038693 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.039215 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.040316 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.040539 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.040644 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.043145 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:15.043148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:15.043151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:15.043176 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:15.044686 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:15.045192 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:15.045197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:15.045218 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:15.045297 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:15.045676 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:15.045681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:15.045683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:15.045914 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:15.046028 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:15.046630 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 14:28:15.046724 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:28:15.047880 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.035544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:43211)
0620 14:28:15.036015 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:15.036053 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:15.037329 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:15.037815 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:15.037822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:15.038133 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:15.038555 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.038568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.039341 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.039349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.040196 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.040205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.040677 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.040681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.040884 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.041503 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:15.041523 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:15.042997 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:15.044801 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:15.044807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:15.044819 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:15.045072 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:15.045417 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:15.045420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:15.045422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:15.045578 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:15.046074 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:15.046081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:15.046368 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 14:28:15.046586 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 14:28:15.046920 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":165,"threads_started":1}
I20260620 14:28:15.048878 14471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45975
I20260620 14:28:15.049116 14471 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:15.049569 14471 heartbeater.cc:511] Master 127.0.98.62:45975 requested a full tablet report, sending...
I20260620 14:28:15.050508 14146 ts_manager.cc:194] Registered new tserver with Master: 5fbc961a9ac2442fa66df3a23bd39ae3 (127.0.98.2:43667)
I20260620 14:28:15.051054 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:28:15.059403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45975
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:15.161638 14476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:15.161834 14476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:15.161871 14476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:15.165709 14476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:15.165753 14476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:15.165807 14476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:15.170519 14476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:15.171286 14476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:15.172112 14476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:15.180660 14482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:15.180670 14484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:15.180711 14481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:15.180976 14476 server_base.cc:1034] running on GCE node
I20260620 14:28:15.181393 14476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:15.181895 14476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:15.183069 14476 hybrid_clock.cc:648] HybridClock initialized: now 1781965695183060 us; error 25 us; skew 500 ppm
Jun 20 14:28:15 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:15.185603 14476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:15.186442 14476 webserver.cc:460] Webserver started at http://127.0.98.3:38891/ using document root <none> and password file <none>
I20260620 14:28:15.187096 14476 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:15.187197 14476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:15.187388 14476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:15.189177 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "d66e7e007aa64175ae7cdf8fc18d50f2"
format_stamp: "Formatted at 2026-06-20 14:28:15 on dist-test-slave-pk3j"
server_key: "2508863621726c4a356532c2ab14e41e"
server_key_iv: "d62cb6a37331d8f347e203ba9459f405"
server_key_version: "encryptionkey@0"
I20260620 14:28:15.189808 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d66e7e007aa64175ae7cdf8fc18d50f2"
format_stamp: "Formatted at 2026-06-20 14:28:15 on dist-test-slave-pk3j"
server_key: "2508863621726c4a356532c2ab14e41e"
server_key_iv: "d62cb6a37331d8f347e203ba9459f405"
server_key_version: "encryptionkey@0"
I20260620 14:28:15.193186 14476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:15.195302 14491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:15.196347 14476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:15.196481 14476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "d66e7e007aa64175ae7cdf8fc18d50f2"
format_stamp: "Formatted at 2026-06-20 14:28:15 on dist-test-slave-pk3j"
server_key: "2508863621726c4a356532c2ab14e41e"
server_key_iv: "d62cb6a37331d8f347e203ba9459f405"
server_key_version: "encryptionkey@0"
I20260620 14:28:15.196605 14476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:15.206900 14476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:15.209909 14476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:15.210101 14476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:15.210585 14476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:15.211303 14476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:15.211370 14476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:15.211434 14476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:15.211481 14476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:15.219779 14476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37733
I20260620 14:28:15.219843 14604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37733 every 8 connection(s)
I20260620 14:28:15.220582 14476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:28:15.224246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 14476
I20260620 14:28:15.224371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:28:15.224592 392 external_mini_cluster.cc:1428] Setting key 0f22ac1c0b5846601f4f18e8813ece34
Jun 20 14:28:15 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:15.233552 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.222327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33723 (local address 127.0.98.62:45975)
0620 14:28:15.222450 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:15.222454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:15.223175 (+ 721us) server_negotiation.cc:407] Connection header received
0620 14:28:15.224390 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:15.224394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:15.224458 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:15.224538 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:15.225958 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.226596 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.227649 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.227853 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.227926 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.230144 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:15.230147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:15.230149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:15.230172 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:15.231597 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:15.232090 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:15.232093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:15.232094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:15.232160 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:15.232438 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:15.232440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:15.232441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:15.232697 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:15.232823 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:15.233352 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 14:28:15.233439 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:28:15.234347 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.222644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:33723)
0620 14:28:15.223035 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:15.223070 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:15.224191 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:15.224740 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:15.224746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:15.225185 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:15.225800 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.225817 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.226713 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.226716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.227541 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.227549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.227972 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.227974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.228096 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.228566 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:15.228581 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:15.230002 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:15.231688 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:15.231692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:15.231701 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:15.232008 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:15.232222 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:15.232224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:15.232225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:15.232355 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:15.232831 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:15.232836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:15.233075 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 14:28:15.233307 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 14:28:15.233525 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
I20260620 14:28:15.235387 14605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45975
I20260620 14:28:15.235597 14605 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:15.236020 14605 heartbeater.cc:511] Master 127.0.98.62:45975 requested a full tablet report, sending...
I20260620 14:28:15.236981 14146 ts_manager.cc:194] Registered new tserver with Master: d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733)
I20260620 14:28:15.237661 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33723
I20260620 14:28:15.238098 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:28:15.246704 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.240148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.0.98.62:45975)
0620 14:28:15.240258 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:15.240262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:15.240441 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:28:15.240530 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:15.240532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:15.240564 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:15.240626 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:15.241321 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.241799 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.242669 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.242879 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.242955 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.244103 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:15.244105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:15.244107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:15.244126 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:15.245498 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:15.245843 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:15.245845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:15.245846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:15.245907 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:15.246112 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:15.246114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:15.246115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:15.246281 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:15.246375 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:15.246535 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:28:15.246609 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:28:15.249148 14146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:28:15.251988 14146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:28:15.265514 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.260305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.0.98.3:37733)
0620 14:28:15.260703 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:15.260714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:15.260946 (+ 232us) server_negotiation.cc:407] Connection header received
0620 14:28:15.261059 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:15.261065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:15.261263 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:15.261392 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:15.261576 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.262415 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.264015 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.264760 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.264873 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.265012 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:15.265288 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 14:28:15.265371 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":139,"threads_started":1}
I20260620 14:28:15.265697 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.259564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43667 (local address 127.0.0.1:52290)
0620 14:28:15.260668 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:15.260680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:15.260757 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:15.261789 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:15.261792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:15.261808 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:15.261883 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.261888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.263305 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.263308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.264331 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.264338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.265157 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.265161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.265345 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.265404 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 14:28:15.265476 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:28:15.265577 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":77,"thread_start_us":81,"threads_started":1}
I20260620 14:28:15.265825 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.260315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.0.98.1:44607)
0620 14:28:15.260739 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:15.260750 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:15.260771 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:28:15.260845 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:15.260852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:15.260996 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:15.261147 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:15.261432 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.262402 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.264156 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.264900 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.264988 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.265132 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:15.265576 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 14:28:15.265661 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":136,"threads_started":1}
I20260620 14:28:15.265913 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.259964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37733 (local address 127.0.0.1:43660)
0620 14:28:15.260830 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:15.260842 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:15.260957 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:15.261385 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:15.261387 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:15.261406 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:15.261488 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.261493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.262538 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.262544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.263894 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.263902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.264886 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.264890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.264990 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.265031 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:28:15.265127 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:28:15.265688 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":146,"thread_start_us":82,"threads_started":1}
I20260620 14:28:15.266145 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.259749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44607 (local address 127.0.0.1:42772)
0620 14:28:15.260378 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:15.260422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:15.260536 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:15.261148 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:15.261153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:15.261173 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:15.261307 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.261313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.262551 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.262555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.264039 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.264049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.265180 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.265183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.265342 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.265399 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 14:28:15.265477 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:28:15.266020 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":603,"thread_start_us":86,"threads_started":1}
I20260620 14:28:15.266340 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.259734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.0.98.2:43667)
0620 14:28:15.261192 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:15.261222 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:15.261243 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:28:15.261313 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:15.261321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:15.261506 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:15.261641 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:15.262011 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.263144 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.264439 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.265031 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.265142 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.265289 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:15.265519 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 14:28:15.266211 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1367,"thread_start_us":103,"threads_started":1}
I20260620 14:28:15.268522 14405 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1e66d8256b4e44b3313b9818b26d11 (DEFAULT_TABLE table=test-table [id=1971b4c5822c44b4bc155604e31e3c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:15.268605 14539 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1e66d8256b4e44b3313b9818b26d11 (DEFAULT_TABLE table=test-table [id=1971b4c5822c44b4bc155604e31e3c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:15.268795 14271 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1e66d8256b4e44b3313b9818b26d11 (DEFAULT_TABLE table=test-table [id=1971b4c5822c44b4bc155604e31e3c58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:15.269536 14405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1e66d8256b4e44b3313b9818b26d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:15.269639 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1e66d8256b4e44b3313b9818b26d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:15.269634 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1e66d8256b4e44b3313b9818b26d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:15.275286 14622 tablet_bootstrap.cc:492] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: Bootstrap starting.
I20260620 14:28:15.276655 14623 tablet_bootstrap.cc:492] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: Bootstrap starting.
I20260620 14:28:15.276655 14624 tablet_bootstrap.cc:492] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Bootstrap starting.
I20260620 14:28:15.277951 14622 tablet_bootstrap.cc:654] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:15.278723 14623 tablet_bootstrap.cc:654] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:15.278836 14622 log.cc:824] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:15.279100 14624 tablet_bootstrap.cc:654] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:15.279953 14624 log.cc:824] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:15.279953 14623 log.cc:824] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:15.280622 14622 tablet_bootstrap.cc:492] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: No bootstrap required, opened a new log
I20260620 14:28:15.280797 14622 ts_tablet_manager.cc:1397] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:28:15.281615 14623 tablet_bootstrap.cc:492] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: No bootstrap required, opened a new log
I20260620 14:28:15.281797 14623 ts_tablet_manager.cc:1397] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:28:15.282346 14624 tablet_bootstrap.cc:492] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: No bootstrap required, opened a new log
I20260620 14:28:15.282511 14624 ts_tablet_manager.cc:1397] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:28:15.283914 14622 raft_consensus.cc:348] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:15.284186 14622 raft_consensus.cc:374] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:15.284240 14622 raft_consensus.cc:729] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fbc961a9ac2442fa66df3a23bd39ae3, State: Initialized, Role: FOLLOWER
I20260620 14:28:15.284659 14622 consensus_queue.cc:260] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:15.285583 14471 heartbeater.cc:503] Master 127.0.98.62:45975 was elected leader, sending a full tablet report...
I20260620 14:28:15.285429 14623 raft_consensus.cc:348] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:15.285604 14622 ts_tablet_manager.cc:1428] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:28:15.285660 14623 raft_consensus.cc:374] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:15.285715 14623 raft_consensus.cc:729] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e7e007aa64175ae7cdf8fc18d50f2, State: Initialized, Role: FOLLOWER
I20260620 14:28:15.286094 14623 consensus_queue.cc:260] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:15.286684 14605 heartbeater.cc:503] Master 127.0.98.62:45975 was elected leader, sending a full tablet report...
I20260620 14:28:15.287128 14624 raft_consensus.cc:348] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:15.287384 14624 raft_consensus.cc:374] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:15.287434 14624 raft_consensus.cc:729] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003b0d6e179a459e8dbd975710b76112, State: Initialized, Role: FOLLOWER
I20260620 14:28:15.287767 14623 ts_tablet_manager.cc:1428] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:28:15.287840 14624 consensus_queue.cc:260] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:15.288493 14337 heartbeater.cc:503] Master 127.0.98.62:45975 was elected leader, sending a full tablet report...
I20260620 14:28:15.288717 14624 ts_tablet_manager.cc:1428] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 14:28:15.344347 14338 tablet.cc:2369] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:28:15.471585 14606 tablet.cc:2369] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:15.488869 14630 raft_consensus.cc:482] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:15.489037 14630 raft_consensus.cc:504] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:15.490211 14630 leader_election.cc:288] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fbc961a9ac2442fa66df3a23bd39ae3 (127.0.98.2:43667), d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733)
I20260620 14:28:15.494870 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.490551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53641 (local address 127.0.98.3:37733)
0620 14:28:15.490744 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:15.490749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:15.491020 (+ 271us) server_negotiation.cc:407] Connection header received
0620 14:28:15.491242 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:15.491246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:15.491295 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:15.491356 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:15.491684 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.492213 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.493575 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.494177 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.494286 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.494347 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:15.494690 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 14:28:15.494752 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 14:28:15.494870 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.490695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44067 (local address 127.0.98.2:43667)
0620 14:28:15.490854 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:15.490858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:15.490872 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:28:15.491010 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:15.491013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:15.491066 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:15.491165 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:15.491432 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.491948 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.493547 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.494177 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.494286 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.494350 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:15.494690 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 14:28:15.494757 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:28:15.494953 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.490438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37733 (local address 127.0.98.1:53641)
0620 14:28:15.490936 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:15.490950 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:15.491091 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:15.491493 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:15.491494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:15.491507 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:15.491583 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.491586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.492316 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.492319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.493470 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.493481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.494298 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.494302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.494465 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.494575 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 14:28:15.494664 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:28:15.494770 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":140,"thread_start_us":86,"threads_started":1}
I20260620 14:28:15.495160 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.490393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43667 (local address 127.0.98.1:44067)
0620 14:28:15.490743 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:15.490777 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:15.490879 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:15.491189 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:15.491193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:15.491214 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:15.491345 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.491353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.492059 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.492066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.493408 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:15.493419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.494298 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.494302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:15.494500 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.494575 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 14:28:15.494664 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:28:15.494772 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":111,"thread_start_us":84,"threads_started":1}
I20260620 14:28:15.495605 14559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1e66d8256b4e44b3313b9818b26d11" candidate_uuid: "003b0d6e179a459e8dbd975710b76112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" is_pre_election: true
I20260620 14:28:15.495795 14425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1e66d8256b4e44b3313b9818b26d11" candidate_uuid: "003b0d6e179a459e8dbd975710b76112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" is_pre_election: true
I20260620 14:28:15.495923 14559 raft_consensus.cc:2454] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 003b0d6e179a459e8dbd975710b76112 in term 0.
I20260620 14:28:15.496075 14425 raft_consensus.cc:2454] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 003b0d6e179a459e8dbd975710b76112 in term 0.
I20260620 14:28:15.496421 14225 leader_election.cc:302] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003b0d6e179a459e8dbd975710b76112, d66e7e007aa64175ae7cdf8fc18d50f2; no voters:
I20260620 14:28:15.496677 14630 raft_consensus.cc:2781] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:28:15.496769 14630 raft_consensus.cc:482] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:15.496830 14630 raft_consensus.cc:3037] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:15.497839 14630 raft_consensus.cc:504] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:15.498154 14630 leader_election.cc:288] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [CANDIDATE]: Term 1 election: Requested vote from peers 5fbc961a9ac2442fa66df3a23bd39ae3 (127.0.98.2:43667), d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733)
I20260620 14:28:15.498485 14559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1e66d8256b4e44b3313b9818b26d11" candidate_uuid: "003b0d6e179a459e8dbd975710b76112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2"
I20260620 14:28:15.498485 14425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1e66d8256b4e44b3313b9818b26d11" candidate_uuid: "003b0d6e179a459e8dbd975710b76112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3"
I20260620 14:28:15.498600 14425 raft_consensus.cc:3037] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:15.498600 14559 raft_consensus.cc:3037] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:15.499835 14559 raft_consensus.cc:2454] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 003b0d6e179a459e8dbd975710b76112 in term 1.
I20260620 14:28:15.499835 14425 raft_consensus.cc:2454] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 003b0d6e179a459e8dbd975710b76112 in term 1.
I20260620 14:28:15.500095 14226 leader_election.cc:302] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003b0d6e179a459e8dbd975710b76112, 5fbc961a9ac2442fa66df3a23bd39ae3; no voters:
I20260620 14:28:15.500257 14630 raft_consensus.cc:2781] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:15.500453 14630 raft_consensus.cc:686] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 1 LEADER]: Becoming Leader. State: Replica: 003b0d6e179a459e8dbd975710b76112, State: Running, Role: LEADER
I20260620 14:28:15.500753 14630 consensus_queue.cc:237] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:15.503444 14145 catalog_manager.cc:5495] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 reported cstate change: term changed from 0 to 1, leader changed from <none> to 003b0d6e179a459e8dbd975710b76112 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "003b0d6e179a459e8dbd975710b76112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } health_report { overall_health: UNKNOWN } } }
W20260620 14:28:15.534469 14472 tablet.cc:2369] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:15.555333 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.552008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.0.98.1:44607)
0620 14:28:15.552140 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:15.552144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:15.552163 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:28:15.552221 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:15.552223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:15.552269 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:15.552377 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:15.552596 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.553064 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.554003 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:15.554226 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:15.554331 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:15.554624 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:15.554986 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:28:15.555117 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:15.555155 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 14:28:15.555206 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:28:15.560811 14425 raft_consensus.cc:1261] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Refusing update from remote peer 003b0d6e179a459e8dbd975710b76112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:15.560832 14559 raft_consensus.cc:1261] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2 [term 1 FOLLOWER]: Refusing update from remote peer 003b0d6e179a459e8dbd975710b76112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:15.561556 14630 consensus_queue.cc:1035] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:15.561779 14633 consensus_queue.cc:1035] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:15.571043 14638 mvcc.cc:205] Tried to move back new op lower bound from 7298931489010061312 to 7298931488773234688. Current Snapshot: MvccSnapshot[applied={T|T < 7298931489010061312}]
I20260620 14:28:15.573035 14636 mvcc.cc:205] Tried to move back new op lower bound from 7298931489010061312 to 7298931488773234688. Current Snapshot: MvccSnapshot[applied={T|T < 7298931489010061312 or (T in {7298931489010061312})}]
I20260620 14:28:15.577551 14144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:15.577701 14144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:15.580080 14144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Sending DeleteTablet for 3 replicas of tablet 2a1e66d8256b4e44b3313b9818b26d11
I20260620 14:28:15.580734 14405 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1e66d8256b4e44b3313b9818b26d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52290
I20260620 14:28:15.580928 14539 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1e66d8256b4e44b3313b9818b26d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43660
I20260620 14:28:15.581176 14645 tablet_replica.cc:330] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: stopping tablet replica
I20260620 14:28:15.581514 14645 raft_consensus.cc:2229] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:15.581497 14271 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1e66d8256b4e44b3313b9818b26d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42772
I20260620 14:28:15.581787 14645 raft_consensus.cc:2258] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:15.582015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 14208
I20260620 14:28:15.581972 14647 tablet_replica.cc:330] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: stopping tablet replica
I20260620 14:28:15.582136 14646 tablet_replica.cc:330] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: stopping tablet replica
I20260620 14:28:15.582372 14646 raft_consensus.cc:2229] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:15.582615 14646 raft_consensus.cc:2258] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:15.584093 14646 ts_tablet_manager.cc:1905] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:15.584474 14645 ts_tablet_manager.cc:1905] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:15.586757 14646 ts_tablet_manager.cc:1918] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:15.586891 14646 log.cc:1196] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/2a1e66d8256b4e44b3313b9818b26d11
I20260620 14:28:15.587116 14645 ts_tablet_manager.cc:1918] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:15.587210 14645 log.cc:1196] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/2a1e66d8256b4e44b3313b9818b26d11
I20260620 14:28:15.587244 14646 ts_tablet_manager.cc:1939] T 2a1e66d8256b4e44b3313b9818b26d11 P d66e7e007aa64175ae7cdf8fc18d50f2: Deleting consensus metadata
I20260620 14:28:15.587529 14645 ts_tablet_manager.cc:1939] T 2a1e66d8256b4e44b3313b9818b26d11 P 5fbc961a9ac2442fa66df3a23bd39ae3: Deleting consensus metadata
I20260620 14:28:15.588475 14131 catalog_manager.cc:4841] TS d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733): tablet 2a1e66d8256b4e44b3313b9818b26d11 (table test-table [id=1971b4c5822c44b4bc155604e31e3c58]) successfully deleted
I20260620 14:28:15.588475 14132 catalog_manager.cc:4841] TS 5fbc961a9ac2442fa66df3a23bd39ae3 (127.0.98.2:43667): tablet 2a1e66d8256b4e44b3313b9818b26d11 (table test-table [id=1971b4c5822c44b4bc155604e31e3c58]) successfully deleted
W20260620 14:28:15.589246 14130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44607 (error 108)
I20260620 14:28:15.589988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 14342
I20260620 14:28:15.590193 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.589934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44607 (local address 127.0.0.1:42788)
0620 14:28:15.590045 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:15.590110 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:28:15.590618 14130 catalog_manager.cc:4584] TS 003b0d6e179a459e8dbd975710b76112 (127.0.98.1:44607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a1e66d8256b4e44b3313b9818b26d11: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44607: connect: Connection refused (error 111)
I20260620 14:28:15.595504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 14476
I20260620 14:28:15.600749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 14114
I20260620 14:28:15.606460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45975
--webserver_interface=127.0.98.62
--webserver_port=38323
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:15.708280 14649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:15.708477 14649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:15.708539 14649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:15.712369 14649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:15.712415 14649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:15.712433 14649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:15.712488 14649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:15.712509 14649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:15.717353 14649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:15.718209 14649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:15.719053 14649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:15.730777 14656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:15.730818 14658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:15.730969 14649 server_base.cc:1034] running on GCE node
W20260620 14:28:15.731079 14655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:15.731633 14649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:15.732571 14649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:15.733723 14649 hybrid_clock.cc:648] HybridClock initialized: now 1781965695733704 us; error 34 us; skew 500 ppm
Jun 20 14:28:15 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:15.736378 14649 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:15.737144 14649 webserver.cc:460] Webserver started at http://127.0.98.62:38323/ using document root <none> and password file <none>
I20260620 14:28:15.737679 14649 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:15.737777 14649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:15.741488 14649 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 14:28:15.743394 14665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:15.744470 14649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:15.744601 14649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "db791c82f21d40f09440c03474b98e71"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "5d63bcecf87e82d5bee0ba10a6686909"
server_key_iv: "35d7a115ec66065da7efaf542d1e90c6"
server_key_version: "encryptionkey@0"
I20260620 14:28:15.745126 14649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:15.773036 14649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:15.776146 14649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:15.776371 14649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:15.783048 14649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45975
I20260620 14:28:15.783295 14717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45975 every 8 connection(s)
I20260620 14:28:15.784085 14649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:15.788157 14718 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:28:15.789678 14718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Bootstrap starting.
I20260620 14:28:15.791674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 14649
I20260620 14:28:15.792093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44607
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45975
--webserver_port=38687
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:15.811662 14718 log.cc:824] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:15.822175 14718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:15.822556 14718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Bootstrap complete.
I20260620 14:28:15.827733 14718 raft_consensus.cc:348] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:15.828217 14718 raft_consensus.cc:729] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db791c82f21d40f09440c03474b98e71, State: Initialized, Role: FOLLOWER
I20260620 14:28:15.828716 14718 consensus_queue.cc:260] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:15.828845 14718 raft_consensus.cc:388] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:15.828910 14718 raft_consensus.cc:482] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:15.829008 14718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:28:15.831260 14718 raft_consensus.cc:504] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:15.831599 14718 leader_election.cc:302] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db791c82f21d40f09440c03474b98e71; no voters:
I20260620 14:28:15.831909 14718 leader_election.cc:288] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:28:15.832023 14722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:28:15.832312 14722 raft_consensus.cc:686] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 2 LEADER]: Becoming Leader. State: Replica: db791c82f21d40f09440c03474b98e71, State: Running, Role: LEADER
I20260620 14:28:15.832604 14722 consensus_queue.cc:237] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:15.833088 14718 sys_catalog.cc:564] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:15.833474 14724 sys_catalog.cc:455] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db791c82f21d40f09440c03474b98e71. Latest consensus state: current_term: 2 leader_uuid: "db791c82f21d40f09440c03474b98e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } } }
I20260620 14:28:15.833618 14724 sys_catalog.cc:458] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:15.833871 14723 sys_catalog.cc:455] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "db791c82f21d40f09440c03474b98e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } } }
I20260620 14:28:15.833947 14723 sys_catalog.cc:458] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:15.834018 14726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:15.838465 14726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:15.839265 14726 catalog_manager.cc:1259] Loaded cluster ID: 8ea5eda6f33840d8a02dc0e9e52a86d4
I20260620 14:28:15.839325 14726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:15.840689 14726 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:28:15.845563 14726 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:28:15.850594 14726 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:15.851244 14726 catalog_manager.cc:5879] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Loaded TSK: 0
I20260620 14:28:15.852319 14726 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:28:15.913591 14720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:15.913832 14720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:15.913904 14720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:15.917824 14720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:15.917889 14720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:15.917972 14720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:15.922786 14720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38687
--tserver_master_addrs=127.0.98.62:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:15.923712 14720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:15.924643 14720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:15.934767 14746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:15.934904 14720 server_base.cc:1034] running on GCE node
W20260620 14:28:15.934762 14745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:15.935031 14748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:15.935508 14720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:15.936195 14720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:15.937379 14720 hybrid_clock.cc:648] HybridClock initialized: now 1781965695937346 us; error 47 us; skew 500 ppm
Jun 20 14:28:15 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:15.940685 14720 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:15.941705 14720 webserver.cc:460] Webserver started at http://127.0.98.1:38687/ using document root <none> and password file <none>
I20260620 14:28:15.942384 14720 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:15.942488 14720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:15.947284 14720 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:28:15.949844 14755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:15.951298 14720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:15.951432 14720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "003b0d6e179a459e8dbd975710b76112"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "7f60dfb9aada4015f8c6a56a36cd8f65"
server_key_iv: "a4a5aecaf46c223b85ac865ce74239b4"
server_key_version: "encryptionkey@0"
I20260620 14:28:15.952077 14720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:15.972007 14720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:15.975385 14720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:15.975557 14720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:15.976263 14720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:15.977669 14762 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:28:15.981571 14720 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:28:15.981649 14720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 14:28:15.981703 14720 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:28:15.983593 14720 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:28:15.983647 14720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 14:28:15.983964 14762 tablet_bootstrap.cc:492] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Bootstrap starting.
I20260620 14:28:15.993817 14720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44607
I20260620 14:28:15.993963 14869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44607 every 8 connection(s)
I20260620 14:28:15.994745 14720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:15.998966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 14720
I20260620 14:28:15.999454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43667
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45975
--webserver_port=36607
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:16.002523 14762 log.cc:824] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:28:16 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:16.010011 14762 tablet_bootstrap.cc:492] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:16.010447 14762 tablet_bootstrap.cc:492] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Bootstrap complete.
I20260620 14:28:16.011660 14762 ts_tablet_manager.cc:1397] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260620 14:28:16.012987 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.996729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:42809)
0620 14:28:15.997314 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:15.997360 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:15.998117 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:15.999144 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:15.999151 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:15.999627 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:16.000095 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.000107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.001723 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.001726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.002997 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.003006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.003900 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.003907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.004149 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.004873 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:16.004899 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:16.006824 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:16.009564 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:16.009570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:16.009580 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:16.009898 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:16.010347 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:16.010354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:16.010359 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:16.010531 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:16.011286 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:16.011295 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:16.011617 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 14:28:16.011807 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 14:28:16.012130 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":142,"threads_started":1}
I20260620 14:28:16.014556 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:15.996688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42809 (local address 127.0.98.62:45975)
0620 14:28:15.997022 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:15.997037 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:15.997479 (+ 442us) server_negotiation.cc:407] Connection header received
0620 14:28:15.998500 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:15.998510 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:15.998773 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:15.999093 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:16.000270 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.001557 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.003129 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.003735 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.003898 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.007039 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:16.007050 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:16.007068 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:16.007110 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:16.009439 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:16.010052 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:16.010061 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:16.010069 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:16.010231 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:16.010626 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:16.010631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:16.010635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:16.011150 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:16.011308 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.013552 (+ 2244us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.013810 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":125,"threads_started":1}
I20260620 14:28:16.014379 14762 raft_consensus.cc:348] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:16.014889 14762 raft_consensus.cc:729] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003b0d6e179a459e8dbd975710b76112, State: Initialized, Role: FOLLOWER
I20260620 14:28:16.015347 14762 consensus_queue.cc:260] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } } peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } }
I20260620 14:28:16.016258 14762 ts_tablet_manager.cc:1428] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:28:16.016911 14870 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45975
I20260620 14:28:16.017127 14870 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:16.017572 14870 heartbeater.cc:511] Master 127.0.98.62:45975 requested a full tablet report, sending...
I20260620 14:28:16.019627 14682 ts_manager.cc:194] Registered new tserver with Master: 003b0d6e179a459e8dbd975710b76112 (127.0.98.1:44607)
I20260620 14:28:16.023167 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42809
I20260620 14:28:16.025657 14870 heartbeater.cc:503] Master 127.0.98.62:45975 was elected leader, sending a full tablet report...
Jun 20 14:28:16 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:28:16.031754 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.022210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44607 (local address 127.0.0.1:42792)
0620 14:28:16.022571 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:16.022614 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:16.022724 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:16.023355 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:16.023361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:16.023727 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:16.023854 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.023861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.024842 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.024846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.025489 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.025499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.026228 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.026244 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.026343 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.026897 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:16.026911 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:16.028295 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:16.030099 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:16.030108 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:16.030113 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:16.030324 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:16.030637 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:16.030641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:16.030643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:16.030740 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:16.031164 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:16.031169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:16.031286 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 14:28:16.031418 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 14:28:16.031591 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
I20260620 14:28:16.031801 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.022299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.0.98.1:44607)
0620 14:28:16.022867 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:16.022876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:16.022896 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:28:16.022960 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:16.022965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:16.023130 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:16.023266 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:16.023973 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.024731 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.025600 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.026022 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.026113 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.028479 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:16.028487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:16.028495 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:16.028514 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:16.029991 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:16.030435 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:16.030449 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:16.030450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:16.030520 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:16.030838 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:16.030841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:16.030843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:16.031052 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:16.031185 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.031513 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.031653 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":82,"threads_started":1}
I20260620 14:28:16.033357 14804 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1e66d8256b4e44b3313b9818b26d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42792
I20260620 14:28:16.034314 14879 tablet_replica.cc:330] stopping tablet replica
I20260620 14:28:16.034701 14879 raft_consensus.cc:2229] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:16.034931 14879 raft_consensus.cc:2258] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:16.035899 14879 ts_tablet_manager.cc:1905] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:16.040140 14879 ts_tablet_manager.cc:1918] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:16.040231 14879 log.cc:1196] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/2a1e66d8256b4e44b3313b9818b26d11
I20260620 14:28:16.040486 14879 ts_tablet_manager.cc:1939] T 2a1e66d8256b4e44b3313b9818b26d11 P 003b0d6e179a459e8dbd975710b76112: Deleting consensus metadata
I20260620 14:28:16.042218 14666 catalog_manager.cc:4841] TS 003b0d6e179a459e8dbd975710b76112 (127.0.98.1:44607): tablet 2a1e66d8256b4e44b3313b9818b26d11 (table test-table [id=1971b4c5822c44b4bc155604e31e3c58]) successfully deleted
W20260620 14:28:16.112825 14874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:16.113029 14874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:16.113068 14874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:16.116844 14874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:16.116891 14874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:16.116945 14874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:16.121824 14874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36607
--tserver_master_addrs=127.0.98.62:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:16.122663 14874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:16.123488 14874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:16.132238 14885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:16.132344 14874 server_base.cc:1034] running on GCE node
W20260620 14:28:16.132243 14884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:16.132248 14887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:16.132831 14874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:16.133459 14874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:16.134668 14874 hybrid_clock.cc:648] HybridClock initialized: now 1781965696134629 us; error 49 us; skew 500 ppm
Jun 20 14:28:16 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:16.137305 14874 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:16.138034 14874 webserver.cc:460] Webserver started at http://127.0.98.2:36607/ using document root <none> and password file <none>
I20260620 14:28:16.138540 14874 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:16.138636 14874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:16.141897 14874 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:28:16.143705 14894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:16.144783 14874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:16.144917 14874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "5fbc961a9ac2442fa66df3a23bd39ae3"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "5da288b13e09210e304fbc2e09ddd732"
server_key_iv: "c0312be85d13f6f04bc65fe687fd6013"
server_key_version: "encryptionkey@0"
I20260620 14:28:16.145418 14874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:16.167510 14874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:16.170467 14874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:16.170653 14874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:16.171191 14874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:16.171945 14874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:16.172015 14874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:16.172084 14874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:16.172127 14874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:16.180785 14874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43667
I20260620 14:28:16.180820 15007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43667 every 8 connection(s)
I20260620 14:28:16.181646 14874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:16.186815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 14874
I20260620 14:28:16.187197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37733
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45975
--webserver_port=38891
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:28:16 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:16.194849 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.183256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44769 (local address 127.0.98.62:45975)
0620 14:28:16.183385 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:16.183388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:16.184044 (+ 656us) server_negotiation.cc:407] Connection header received
0620 14:28:16.185270 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:16.185273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:16.185329 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:16.185430 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:16.186441 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.187044 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.188219 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.188487 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.188606 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.191168 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:16.191171 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:16.191173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:16.191195 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:16.192628 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:16.193278 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:16.193287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:16.193288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:16.193362 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:16.193775 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:16.193777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:16.193778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:16.193958 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:16.194032 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.194627 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.194722 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:28:16.195911 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.183520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:44769)
0620 14:28:16.183899 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:16.183944 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:16.185052 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:16.185597 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:16.185604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:16.185897 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:16.186294 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.186306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.187163 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.187167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.188095 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.188103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.188633 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.188637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.188795 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.189325 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:16.189339 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:16.191009 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:16.192750 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:16.192759 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:16.192774 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:16.193137 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:16.193462 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:16.193468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:16.193472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:16.193673 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:16.194064 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:16.194073 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:16.194371 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 14:28:16.194599 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 14:28:16.194923 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":98,"threads_started":1}
I20260620 14:28:16.197088 15008 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45975
I20260620 14:28:16.197350 15008 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:16.197921 15008 heartbeater.cc:511] Master 127.0.98.62:45975 requested a full tablet report, sending...
I20260620 14:28:16.199000 14682 ts_manager.cc:194] Registered new tserver with Master: 5fbc961a9ac2442fa66df3a23bd39ae3 (127.0.98.2:43667)
I20260620 14:28:16.199731 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44769
W20260620 14:28:16.294732 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:16.294945 15011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:16.294981 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:16.298734 15011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:16.298779 15011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:16.298830 15011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:16.303608 15011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38891
--tserver_master_addrs=127.0.98.62:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:16.304389 15011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:16.305228 15011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:16.313920 15016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:16.314007 15017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:16.313920 15019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:16.314183 15011 server_base.cc:1034] running on GCE node
I20260620 14:28:16.314538 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:16.315085 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:16.316257 15011 hybrid_clock.cc:648] HybridClock initialized: now 1781965696316228 us; error 40 us; skew 500 ppm
Jun 20 14:28:16 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:16.318783 15011 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:16.319597 15011 webserver.cc:460] Webserver started at http://127.0.98.3:38891/ using document root <none> and password file <none>
I20260620 14:28:16.320102 15011 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:16.320192 15011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:16.323822 15011 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:16.325680 15026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:16.326817 15011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:16.326949 15011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "d66e7e007aa64175ae7cdf8fc18d50f2"
format_stamp: "Formatted at 2026-06-20 14:28:15 on dist-test-slave-pk3j"
server_key: "2508863621726c4a356532c2ab14e41e"
server_key_iv: "d62cb6a37331d8f347e203ba9459f405"
server_key_version: "encryptionkey@0"
I20260620 14:28:16.327430 15011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:16.351416 15011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:16.354017 15011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:16.354192 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:16.354696 15011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:16.355403 15011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:16.355468 15011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:16.355542 15011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:16.355589 15011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:16.363771 15011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37733
I20260620 14:28:16.363795 15139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37733 every 8 connection(s)
I20260620 14:28:16.364655 15011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:28:16 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:16.373639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 15011
I20260620 14:28:16.377225 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.366324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32859 (local address 127.0.98.62:45975)
0620 14:28:16.366455 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:16.366458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:16.367191 (+ 733us) server_negotiation.cc:407] Connection header received
0620 14:28:16.368415 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:16.368418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:16.368474 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:16.368571 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:16.369682 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.370185 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.371264 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.371481 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.371588 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.373955 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:16.373963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:16.373965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:16.373985 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:16.375277 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:16.375774 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:16.375778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:16.375780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:16.375863 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:16.376186 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:16.376191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:16.376192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:16.376358 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:16.376468 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.376935 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.377043 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:28:16.378119 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.366661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:32859)
0620 14:28:16.367056 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:16.367092 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:16.368213 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:16.368748 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:16.368754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:16.369057 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:16.369548 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.369560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.370314 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.370322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.371140 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.371148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.371618 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.371622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.371776 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.372307 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:16.372321 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:16.373803 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:16.375411 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:16.375417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:16.375428 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:16.375671 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:16.375961 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:16.375965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:16.375967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:16.376092 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:16.376467 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:16.376473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:16.376705 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 14:28:16.376880 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 14:28:16.377151 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
I20260620 14:28:16.379172 15140 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45975
I20260620 14:28:16.379415 15140 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:16.379854 15140 heartbeater.cc:511] Master 127.0.98.62:45975 requested a full tablet report, sending...
I20260620 14:28:16.380852 14682 ts_manager.cc:194] Registered new tserver with Master: d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733)
I20260620 14:28:16.381425 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32859
I20260620 14:28:16.381740 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.374226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.0.98.62:45975)
0620 14:28:16.374453 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:16.374457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:16.374635 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:28:16.374770 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:16.374774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:16.374816 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:16.374899 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:16.375546 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.376045 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.376898 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.377123 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.377250 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.378615 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:16.378621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:16.378625 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:16.378655 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:16.380120 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:16.380554 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:16.380557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:16.380558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:16.380637 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:16.380920 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:16.380923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:16.380925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:16.381095 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:16.381324 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.381478 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.381585 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":76,"threads_started":1}
I20260620 14:28:16.383355 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:28:16.386708 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.383899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.0.98.62:45975)
0620 14:28:16.383995 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:16.384005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:16.384094 (+ 89us) server_negotiation.cc:407] Connection header received
0620 14:28:16.384209 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:16.384211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:16.384281 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:16.384351 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:16.384501 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.384919 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.385675 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.385876 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.385950 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.386165 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:16.386443 (+ 278us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:28:16.386520 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.386574 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.386631 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:28:16.387225 14682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:28:16.390197 14682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:28:16.397444 14804 tablet_service.cc:1467] Processing CreateTablet for tablet 7477c9325f0d465ba02ab41a0827dc8f (DEFAULT_TABLE table=test-table [id=6cd2d38295f446cf9f366a964656fd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:16.398036 14804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7477c9325f0d465ba02ab41a0827dc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:16.400746 14762 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Bootstrap starting.
I20260620 14:28:16.401079 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.396062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37733 (local address 127.0.0.1:43670)
0620 14:28:16.396462 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:16.396476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:16.396554 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:16.397018 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:16.397023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:16.397044 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:16.397130 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.397136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.398632 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.398636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.399763 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.399776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.400697 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.400700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.400792 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.400829 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:28:16.400891 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:28:16.400962 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346}
I20260620 14:28:16.401479 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.396157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.0.98.3:37733)
0620 14:28:16.396460 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:16.396473 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:16.396525 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:28:16.396668 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:16.396673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:16.396856 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:16.397318 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:16.397451 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.398521 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.399907 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.400575 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.400695 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.401242 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.401310 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.401383 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":111,"threads_started":1}
I20260620 14:28:16.401813 14762 tablet_bootstrap.cc:654] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:16.402040 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.396406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43667 (local address 127.0.0.1:52306)
0620 14:28:16.396649 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:16.396662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:16.396748 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:16.397828 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:16.397830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:16.397863 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:16.397950 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.397955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.399279 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.399286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.400160 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.400170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.401699 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.401703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.401769 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.401786 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:28:16.401845 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:28:16.401919 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260620 14:28:16.402294 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.396875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.0.98.2:43667)
0620 14:28:16.397217 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:16.397233 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:16.397257 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:28:16.397330 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:16.397335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:16.397523 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:16.397669 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:16.398073 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.399145 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.400286 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.401475 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.401572 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.401690 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.401902 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.402001 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":120,"threads_started":1}
I20260620 14:28:16.403357 14762 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: No bootstrap required, opened a new log
I20260620 14:28:16.403442 14762 ts_tablet_manager.cc:1397] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:28:16.403913 14762 raft_consensus.cc:348] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:16.404007 14762 raft_consensus.cc:374] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:16.404031 14762 raft_consensus.cc:729] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003b0d6e179a459e8dbd975710b76112, State: Initialized, Role: FOLLOWER
I20260620 14:28:16.404114 14762 consensus_queue.cc:260] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:16.404345 14762 ts_tablet_manager.cc:1428] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:28:16.405068 15074 tablet_service.cc:1467] Processing CreateTablet for tablet 7477c9325f0d465ba02ab41a0827dc8f (DEFAULT_TABLE table=test-table [id=6cd2d38295f446cf9f366a964656fd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:16.405066 14942 tablet_service.cc:1467] Processing CreateTablet for tablet 7477c9325f0d465ba02ab41a0827dc8f (DEFAULT_TABLE table=test-table [id=6cd2d38295f446cf9f366a964656fd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:16.406056 14942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7477c9325f0d465ba02ab41a0827dc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:16.406061 15074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7477c9325f0d465ba02ab41a0827dc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:16.413874 15154 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Bootstrap starting.
I20260620 14:28:16.414070 15153 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Bootstrap starting.
I20260620 14:28:16.416368 15153 tablet_bootstrap.cc:654] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:16.416791 15154 tablet_bootstrap.cc:654] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:16.417702 15153 log.cc:824] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:16.417702 15154 log.cc:824] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:16.419696 15153 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: No bootstrap required, opened a new log
I20260620 14:28:16.419696 15154 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: No bootstrap required, opened a new log
I20260620 14:28:16.419888 15154 ts_tablet_manager.cc:1397] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:28:16.419888 15153 ts_tablet_manager.cc:1397] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:28:16.423885 15153 raft_consensus.cc:348] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:16.423890 15154 raft_consensus.cc:348] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:16.424165 15153 raft_consensus.cc:374] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:16.424165 15154 raft_consensus.cc:374] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:16.424233 15154 raft_consensus.cc:729] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fbc961a9ac2442fa66df3a23bd39ae3, State: Initialized, Role: FOLLOWER
I20260620 14:28:16.424233 15153 raft_consensus.cc:729] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e7e007aa64175ae7cdf8fc18d50f2, State: Initialized, Role: FOLLOWER
I20260620 14:28:16.424679 15153 consensus_queue.cc:260] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:16.424679 15154 consensus_queue.cc:260] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:16.425287 15140 heartbeater.cc:503] Master 127.0.98.62:45975 was elected leader, sending a full tablet report...
I20260620 14:28:16.425501 15008 heartbeater.cc:503] Master 127.0.98.62:45975 was elected leader, sending a full tablet report...
I20260620 14:28:16.425633 15154 ts_tablet_manager.cc:1428] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260620 14:28:16.425632 15153 ts_tablet_manager.cc:1428] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260620 14:28:16.432435 15009 tablet.cc:2369] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:28:16.495985 14871 tablet.cc:2369] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:16.611301 15157 raft_consensus.cc:482] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:16.611513 15157 raft_consensus.cc:504] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:16.612716 15157 leader_election.cc:288] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733), 003b0d6e179a459e8dbd975710b76112 (127.0.98.1:44607)
W20260620 14:28:16.615592 15141 tablet.cc:2369] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:16.616482 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.612991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37733 (local address 127.0.98.2:47709)
0620 14:28:16.613136 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:16.613153 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:16.613279 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:16.613626 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:16.613629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:16.613648 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:16.613764 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.613769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.614507 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.614510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.615310 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.615319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.616147 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.616150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.616249 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.616269 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:28:16.616329 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:28:16.616380 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:28:16.616567 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.613070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47709 (local address 127.0.98.3:37733)
0620 14:28:16.613239 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:16.613245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:16.613278 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:28:16.613432 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:16.613437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:16.613503 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:16.613597 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:16.613884 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.614386 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.615447 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.616025 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.616133 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.616195 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.616383 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.616448 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:28:16.617293 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.612994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44607 (local address 127.0.98.2:33443)
0620 14:28:16.613640 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:16.613651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:16.613766 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:16.614067 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:16.614070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:16.614085 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:16.614206 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.614211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.614939 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.614944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.615922 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:16.615932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.616856 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.616860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:16.616957 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.616976 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:28:16.617078 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:28:16.617141 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":284,"thread_start_us":155,"threads_started":1}
I20260620 14:28:16.617332 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.613100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33443 (local address 127.0.98.1:44607)
0620 14:28:16.613426 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:16.613431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:16.613733 (+ 302us) server_negotiation.cc:407] Connection header received
0620 14:28:16.613877 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:16.613880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:16.613948 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:16.614059 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:16.614321 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.614827 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.616056 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.616716 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.616854 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.617022 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.617115 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.617175 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":115,"threads_started":1}
I20260620 14:28:16.617314 15094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7477c9325f0d465ba02ab41a0827dc8f" candidate_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" is_pre_election: true
I20260620 14:28:16.617617 15094 raft_consensus.cc:2454] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fbc961a9ac2442fa66df3a23bd39ae3 in term 0.
I20260620 14:28:16.617801 14824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7477c9325f0d465ba02ab41a0827dc8f" candidate_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003b0d6e179a459e8dbd975710b76112" is_pre_election: true
I20260620 14:28:16.618098 14824 raft_consensus.cc:2454] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fbc961a9ac2442fa66df3a23bd39ae3 in term 0.
I20260620 14:28:16.618101 14896 leader_election.cc:302] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fbc961a9ac2442fa66df3a23bd39ae3, d66e7e007aa64175ae7cdf8fc18d50f2; no voters:
I20260620 14:28:16.618371 15157 raft_consensus.cc:2781] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:28:16.618474 15157 raft_consensus.cc:482] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:16.618530 15157 raft_consensus.cc:3037] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:16.619546 15157 raft_consensus.cc:504] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:16.619848 15157 leader_election.cc:288] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [CANDIDATE]: Term 1 election: Requested vote from peers d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733), 003b0d6e179a459e8dbd975710b76112 (127.0.98.1:44607)
I20260620 14:28:16.620148 15094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7477c9325f0d465ba02ab41a0827dc8f" candidate_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2"
I20260620 14:28:16.620152 14824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7477c9325f0d465ba02ab41a0827dc8f" candidate_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003b0d6e179a459e8dbd975710b76112"
I20260620 14:28:16.620267 15094 raft_consensus.cc:3037] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:16.620267 14824 raft_consensus.cc:3037] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:16.621450 14824 raft_consensus.cc:2454] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fbc961a9ac2442fa66df3a23bd39ae3 in term 1.
I20260620 14:28:16.621454 15094 raft_consensus.cc:2454] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fbc961a9ac2442fa66df3a23bd39ae3 in term 1.
I20260620 14:28:16.621706 14895 leader_election.cc:302] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003b0d6e179a459e8dbd975710b76112, 5fbc961a9ac2442fa66df3a23bd39ae3; no voters:
I20260620 14:28:16.621860 15157 raft_consensus.cc:2781] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:16.622084 15157 raft_consensus.cc:686] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 LEADER]: Becoming Leader. State: Replica: 5fbc961a9ac2442fa66df3a23bd39ae3, State: Running, Role: LEADER
I20260620 14:28:16.622393 15157 consensus_queue.cc:237] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:16.625310 14681 catalog_manager.cc:5495] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fbc961a9ac2442fa66df3a23bd39ae3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } health_report { overall_health: UNKNOWN } } }
I20260620 14:28:16.691244 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.687837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.0.98.2:43667)
0620 14:28:16.687951 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:16.687954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:16.687966 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:28:16.688063 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:16.688065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:16.688118 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:16.688237 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:16.688469 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.688945 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.689820 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:16.690053 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:16.690166 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:16.690461 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:16.690856 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:28:16.690987 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:16.691050 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:28:16.691106 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:28:16.696776 15094 raft_consensus.cc:1261] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 1 FOLLOWER]: Refusing update from remote peer 5fbc961a9ac2442fa66df3a23bd39ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:16.696772 14824 raft_consensus.cc:1261] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Refusing update from remote peer 5fbc961a9ac2442fa66df3a23bd39ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:16.697528 15161 consensus_queue.cc:1035] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:16.697718 15157 consensus_queue.cc:1035] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:16.702992 15164 mvcc.cc:205] Tried to move back new op lower bound from 7298931493662609408 to 7298931493367603200. Current Snapshot: MvccSnapshot[applied={T|T < 7298931493662609408}]
I20260620 14:28:16.704612 15165 mvcc.cc:205] Tried to move back new op lower bound from 7298931493662609408 to 7298931493367603200. Current Snapshot: MvccSnapshot[applied={T|T < 7298931493662609408}]
I20260620 14:28:16.710357 14682 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:16.710501 14682 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:16.712719 14682 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Sending DeleteTablet for 3 replicas of tablet 7477c9325f0d465ba02ab41a0827dc8f
I20260620 14:28:16.713399 14804 tablet_service.cc:1514] Processing DeleteTablet for tablet 7477c9325f0d465ba02ab41a0827dc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42792
I20260620 14:28:16.713387 15074 tablet_service.cc:1514] Processing DeleteTablet for tablet 7477c9325f0d465ba02ab41a0827dc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43670
I20260620 14:28:16.713536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 14720
I20260620 14:28:16.713559 14942 tablet_service.cc:1514] Processing DeleteTablet for tablet 7477c9325f0d465ba02ab41a0827dc8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52306
I20260620 14:28:16.717409 15174 tablet_replica.cc:330] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: stopping tablet replica
I20260620 14:28:16.717747 15174 raft_consensus.cc:2229] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:16.718050 15174 raft_consensus.cc:2258] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:16.718365 15175 tablet_replica.cc:330] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: stopping tablet replica
I20260620 14:28:16.718688 15175 raft_consensus.cc:2229] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:28:16.719120 15175 raft_consensus.cc:2258] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:16.719537 15174 ts_tablet_manager.cc:1905] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:28:16.719764 14666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44607 (error 108)
I20260620 14:28:16.720528 15175 ts_tablet_manager.cc:1905] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:16.720727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 14874
I20260620 14:28:16.721998 15174 ts_tablet_manager.cc:1918] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:16.722111 15174 log.cc:1196] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/7477c9325f0d465ba02ab41a0827dc8f
I20260620 14:28:16.722476 15174 ts_tablet_manager.cc:1939] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Deleting consensus metadata
I20260620 14:28:16.722739 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.722489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44607 (local address 127.0.0.1:42798)
0620 14:28:16.722614 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:16.722680 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:28:16.722949 14666 catalog_manager.cc:4584] TS 003b0d6e179a459e8dbd975710b76112 (127.0.98.1:44607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7477c9325f0d465ba02ab41a0827dc8f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44607: connect: Connection refused (error 111)
I20260620 14:28:16.723634 14667 catalog_manager.cc:4841] TS d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733): tablet 7477c9325f0d465ba02ab41a0827dc8f (table test-table [id=6cd2d38295f446cf9f366a964656fd19]) successfully deleted
I20260620 14:28:16.726783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 15011
I20260620 14:28:16.727277 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:16.726995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43667 (local address 127.0.0.1:52332)
0620 14:28:16.727149 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:16.727201 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 14:28:16.732326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 14649
I20260620 14:28:16.738329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45975
--webserver_interface=127.0.98.62
--webserver_port=38323
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:16.841251 15177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:16.841459 15177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:16.841495 15177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:16.845295 15177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:16.845336 15177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:16.845355 15177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:16.845402 15177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:16.845432 15177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:16.850370 15177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:16.851229 15177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:16.852094 15177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:28:16.859951 15177 server_base.cc:1034] running on GCE node
W20260620 14:28:16.859974 15185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:16.860010 15182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:16.859993 15183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:16.860697 15177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:16.861771 15177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:16.862944 15177 hybrid_clock.cc:648] HybridClock initialized: now 1781965696862928 us; error 29 us; skew 500 ppm
Jun 20 14:28:16 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:16.865530 15177 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:16.866333 15177 webserver.cc:460] Webserver started at http://127.0.98.62:38323/ using document root <none> and password file <none>
I20260620 14:28:16.866835 15177 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:16.866894 15177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:16.870249 15177 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:28:16.872126 15192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:16.873148 15177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 14:28:16.873322 15177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "db791c82f21d40f09440c03474b98e71"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "5d63bcecf87e82d5bee0ba10a6686909"
server_key_iv: "35d7a115ec66065da7efaf542d1e90c6"
server_key_version: "encryptionkey@0"
I20260620 14:28:16.873782 15177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:16.887192 15177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:16.890056 15177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:16.890240 15177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:16.896874 15177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45975
I20260620 14:28:16.897137 15244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45975 every 8 connection(s)
I20260620 14:28:16.897873 15177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:16.901566 15245 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:28:16.902969 15245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Bootstrap starting.
I20260620 14:28:16.904136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 15177
I20260620 14:28:16.904515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44607
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45975
--webserver_port=38687
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:16.919322 15245 log.cc:824] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:16.929960 15245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:16.930294 15245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71: Bootstrap complete.
I20260620 14:28:16.934144 15245 raft_consensus.cc:348] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:16.934514 15245 raft_consensus.cc:729] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: db791c82f21d40f09440c03474b98e71, State: Initialized, Role: FOLLOWER
I20260620 14:28:16.934993 15245 consensus_queue.cc:260] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:16.935139 15245 raft_consensus.cc:388] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:16.935217 15245 raft_consensus.cc:482] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:16.935338 15245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:28:16.937047 15245 raft_consensus.cc:504] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:16.937394 15245 leader_election.cc:302] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db791c82f21d40f09440c03474b98e71; no voters:
I20260620 14:28:16.937680 15245 leader_election.cc:288] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:28:16.937803 15249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:28:16.938064 15249 raft_consensus.cc:686] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [term 3 LEADER]: Becoming Leader. State: Replica: db791c82f21d40f09440c03474b98e71, State: Running, Role: LEADER
I20260620 14:28:16.938472 15249 consensus_queue.cc:237] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } }
I20260620 14:28:16.938982 15245 sys_catalog.cc:564] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:16.939774 15250 sys_catalog.cc:455] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "db791c82f21d40f09440c03474b98e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } } }
I20260620 14:28:16.939886 15250 sys_catalog.cc:458] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:16.940249 15251 sys_catalog.cc:455] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db791c82f21d40f09440c03474b98e71. Latest consensus state: current_term: 3 leader_uuid: "db791c82f21d40f09440c03474b98e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db791c82f21d40f09440c03474b98e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45975 } } }
I20260620 14:28:16.940385 15251 sys_catalog.cc:458] T 00000000000000000000000000000000 P db791c82f21d40f09440c03474b98e71 [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:16.940830 15258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:16.944514 15258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:16.945015 15258 catalog_manager.cc:1259] Loaded cluster ID: 8ea5eda6f33840d8a02dc0e9e52a86d4
I20260620 14:28:16.945088 15258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:28:16.945922 15258 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f73969159ad google::LogMessage::Fail() at ??:0
@ 0x7f739691888a google::LogMessage::SendToLog() at ??:0
@ 0x7f73969154de google::LogMessage::Flush() at ??:0
@ 0x7f7396916a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f73a1254fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f73a1255d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f73a1252579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f73a1296640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7398b673d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7398b67c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7398b693f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7398b58512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f739a19c6db start_thread at ??:0
@ 0x7f739541361f clone at ??:0
W20260620 14:28:17.028365 15247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:17.028599 15247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:17.028681 15247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:17.032568 15247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:17.032644 15247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:17.032733 15247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:17.037674 15247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38687
--tserver_master_addrs=127.0.98.62:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:17.038642 15247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:17.039645 15247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:17.049054 15287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:17.049036 15286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:17.049196 15247 server_base.cc:1034] running on GCE node
W20260620 14:28:17.049114 15289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:17.049782 15247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:17.050345 15247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:17.051525 15247 hybrid_clock.cc:648] HybridClock initialized: now 1781965697051511 us; error 25 us; skew 500 ppm
Jun 20 14:28:17 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:17.054108 15247 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:17.054881 15247 webserver.cc:460] Webserver started at http://127.0.98.1:38687/ using document root <none> and password file <none>
I20260620 14:28:17.055374 15247 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:17.055429 15247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:17.059055 15247 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:28:17.061034 15296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:17.062171 15247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:17.062296 15247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "003b0d6e179a459e8dbd975710b76112"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "7f60dfb9aada4015f8c6a56a36cd8f65"
server_key_iv: "a4a5aecaf46c223b85ac865ce74239b4"
server_key_version: "encryptionkey@0"
I20260620 14:28:17.062779 15247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:17.079257 15247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:17.082293 15247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:17.082471 15247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:17.082983 15247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:17.083947 15303 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:28:17.089061 15247 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:28:17.089130 15247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 14:28:17.089241 15247 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:28:17.090798 15247 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:28:17.090873 15247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 14:28:17.091147 15303 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Bootstrap starting.
I20260620 14:28:17.100916 15247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44607
I20260620 14:28:17.101127 15410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44607 every 8 connection(s)
I20260620 14:28:17.101837 15247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:17.110374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 15247
I20260620 14:28:17.110810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43667
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45975
--webserver_port=36607
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:17.111204 15303 log.cc:824] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:17.118891 15303 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:17.119279 15303 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Bootstrap complete.
I20260620 14:28:17.120661 15303 ts_tablet_manager.cc:1397] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.000s
I20260620 14:28:17.123539 15303 raft_consensus.cc:348] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:17.124046 15303 raft_consensus.cc:729] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003b0d6e179a459e8dbd975710b76112, State: Initialized, Role: FOLLOWER
I20260620 14:28:17.124558 15303 consensus_queue.cc:260] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:17.125604 15303 ts_tablet_manager.cc:1428] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260620 14:28:17.163295 15413 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:28:17.103556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:53097)
0620 14:28:17.103993 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.104040 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:17.104787 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:17.162424 (+ 57637us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":143,"threads_started":1}
W20260620 14:28:17.163777 15411 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45975 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45975: Client connection negotiation failed: client connection to 127.0.98.62:45975: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:28:17.164511 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:17.164251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:59947)
0620 14:28:17.164382 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.164452 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:28:17.165170 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:17.164951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:52935)
0620 14:28:17.165069 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.165117 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 14:28:17.165380 15411 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:28:17.216109 15415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:17.216303 15415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:17.216341 15415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:17.220112 15415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:17.220157 15415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:17.220208 15415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:17.224725 15415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36607
--tserver_master_addrs=127.0.98.62:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:17.225567 15415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:17.226512 15415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:17.235103 15422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:17.235078 15424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:17.235095 15421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:17.235145 15415 server_base.cc:1034] running on GCE node
I20260620 14:28:17.235641 15415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:17.236236 15415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:17.237408 15415 hybrid_clock.cc:648] HybridClock initialized: now 1781965697237393 us; error 28 us; skew 500 ppm
Jun 20 14:28:17 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:17.240106 15415 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:17.240834 15415 webserver.cc:460] Webserver started at http://127.0.98.2:36607/ using document root <none> and password file <none>
I20260620 14:28:17.241336 15415 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:17.241396 15415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:17.244699 15415 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:28:17.246727 15431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:17.247789 15415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:28:17.247910 15415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "5fbc961a9ac2442fa66df3a23bd39ae3"
format_stamp: "Formatted at 2026-06-20 14:28:14 on dist-test-slave-pk3j"
server_key: "5da288b13e09210e304fbc2e09ddd732"
server_key_iv: "c0312be85d13f6f04bc65fe687fd6013"
server_key_version: "encryptionkey@0"
I20260620 14:28:17.248365 15415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:17.260424 15415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:17.263300 15415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:17.263450 15415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:17.264015 15415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:17.264911 15438 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:28:17.268667 15415 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:28:17.268735 15415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 14:28:17.268824 15415 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:28:17.270349 15415 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:28:17.270414 15415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 14:28:17.270622 15438 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Bootstrap starting.
I20260620 14:28:17.275345 15438 log.cc:824] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:17.280839 15415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43667
I20260620 14:28:17.280916 15546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43667 every 8 connection(s)
I20260620 14:28:17.281677 15415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:17.283123 15438 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:17.283479 15438 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Bootstrap complete.
I20260620 14:28:17.283809 15438 ts_tablet_manager.cc:1397] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260620 14:28:17.284613 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:17.283316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:42051)
0620 14:28:17.283740 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.283874 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
W20260620 14:28:17.284979 15547 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45975 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45975: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
I20260620 14:28:17.285597 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:17.285358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:42789)
0620 14:28:17.285466 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.285530 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:28:17.286262 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:17.286046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:55327)
0620 14:28:17.286157 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.286207 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 14:28:17.286448 15547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:28:17.286751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 15415
I20260620 14:28:17.287113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37733
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45975
--webserver_port=38891
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:17.287027 15438 raft_consensus.cc:348] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:17.287497 15438 raft_consensus.cc:729] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fbc961a9ac2442fa66df3a23bd39ae3, State: Initialized, Role: FOLLOWER
I20260620 14:28:17.287920 15438 consensus_queue.cc:260] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:17.288837 15438 ts_tablet_manager.cc:1428] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:28:17.288955 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:17.288694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:52753)
0620 14:28:17.288820 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.288880 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:28:17.289616 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:17.289368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:41347)
0620 14:28:17.289494 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.289558 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:28:17.388775 15551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:17.389005 15551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:17.389092 15551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:17.393026 15551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:17.393102 15551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:17.393185 15551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:17.398175 15551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38891
--tserver_master_addrs=127.0.98.62:45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:17.398977 15551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:17.399822 15551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:17.408020 15556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:17.408058 15557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:17.408051 15559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:17.408165 15551 server_base.cc:1034] running on GCE node
I20260620 14:28:17.408607 15551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:17.409147 15551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:17.410339 15551 hybrid_clock.cc:648] HybridClock initialized: now 1781965697410322 us; error 27 us; skew 500 ppm
Jun 20 14:28:17 dist-test-slave-pk3j krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:17.412865 15551 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:17.413690 15551 webserver.cc:460] Webserver started at http://127.0.98.3:38891/ using document root <none> and password file <none>
I20260620 14:28:17.414175 15551 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:17.414233 15551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:17.417555 15551 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:28:17.419422 15566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:17.420493 15551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:17.420619 15551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "d66e7e007aa64175ae7cdf8fc18d50f2"
format_stamp: "Formatted at 2026-06-20 14:28:15 on dist-test-slave-pk3j"
server_key: "2508863621726c4a356532c2ab14e41e"
server_key_iv: "d62cb6a37331d8f347e203ba9459f405"
server_key_version: "encryptionkey@0"
I20260620 14:28:17.421089 15551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:17.441815 15551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:17.444590 15551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:17.444793 15551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:17.445428 15551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:17.446152 15551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:17.446197 15551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:17.446261 15551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:17.446300 15551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:17.454885 15551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37733
I20260620 14:28:17.454962 15679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37733 every 8 connection(s)
I20260620 14:28:17.455773 15551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:28:17.459138 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:17.457661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:52931)
0620 14:28:17.458461 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.458552 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":515,"threads_started":1}
W20260620 14:28:17.459568 15680 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45975 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45975: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
I20260620 14:28:17.460060 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:17.459882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:60235)
0620 14:28:17.459973 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.460014 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:28:17.460609 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:17.460440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:60839)
0620 14:28:17.460527 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:17.460565 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:28:17.460783 15680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:28:17.463697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 15551
I20260620 14:28:18.166258 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:18.165889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:38013)
0620 14:28:18.166114 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:18.166160 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
I20260620 14:28:18.290526 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:18.290166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:46155)
0620 14:28:18.290399 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:18.290456 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
I20260620 14:28:18.461627 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:18.461277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:41633)
0620 14:28:18.461499 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:18.461546 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260620 14:28:18.588260 15687 raft_consensus.cc:482] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:18.588490 15687 raft_consensus.cc:504] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:18.589653 15687 leader_election.cc:288] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733), 5fbc961a9ac2442fa66df3a23bd39ae3 (127.0.98.2:43667)
Jun 20 14:28:18 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 14:28:18 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 14:28:18.603530 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:18.589869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43667 (local address 127.0.98.1:40737)
0620 14:28:18.590359 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:18.590375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:18.590489 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:18.591427 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:18.591433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:18.592045 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:18.592540 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:18.592553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:18.594124 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:18.594128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:18.595079 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:18.595088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:18.595776 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:18.595780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:18.596026 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:18.596670 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:18.596685 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:18.598732 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:18.600763 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:18.600769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:18.600783 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:18.601223 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:18.601578 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:18.601582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:18.601584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:18.601778 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:18.602457 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:18.602463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:18.602805 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 14:28:18.603018 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 14:28:18.603349 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":299,"spinlock_wait_cycles":24320,"thread_start_us":86,"threads_started":1}
I20260620 14:28:18.603559 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:18.590016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36383 (local address 127.0.98.3:37733)
0620 14:28:18.590354 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:18.590383 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:18.590427 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:28:18.591163 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:18.591169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:18.591554 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:18.591838 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:18.592704 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:18.594019 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:18.595180 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:18.595657 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:18.595851 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:18.598635 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:18.598642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:18.598663 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:18.598687 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:18.600640 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:18.601339 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:18.601344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:18.601349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:18.601482 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:18.601878 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:18.601881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:18.601882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:18.602342 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:18.602469 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:18.603151 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 14:28:18.603397 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":64,"threads_started":1}
I20260620 14:28:18.603578 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:18.589991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40737 (local address 127.0.98.2:43667)
0620 14:28:18.590375 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:18.590385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:18.590471 (+ 86us) server_negotiation.cc:407] Connection header received
0620 14:28:18.590728 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:18.590733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:18.591082 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:18.591313 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:18.592852 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:18.593994 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:18.595226 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:18.595655 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:18.595846 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:18.598879 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:18.598886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:18.598900 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:18.598932 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:18.600640 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:18.601339 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:18.601344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:18.601349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:18.601482 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:18.601878 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:18.601881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:18.601882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:18.602341 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:18.602469 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:18.603151 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 14:28:18.603397 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":62,"threads_started":1}
I20260620 14:28:18.604036 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:18.589869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37733 (local address 127.0.98.1:36383)
0620 14:28:18.590074 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:18.590089 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:18.590183 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:18.591792 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:18.591795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:18.592046 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:18.592538 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:18.592552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:18.594168 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:18.594171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:18.595066 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:18.595075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:18.595780 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:18.595783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:18.596027 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:18.596730 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:18.596742 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:18.598355 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:18.600763 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:18.600769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:18.600783 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:18.601223 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:18.601578 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:18.601582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:18.601584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:18.601778 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:18.602457 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:18.602463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:18.602805 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 14:28:18.603018 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 14:28:18.603931 (+ 913us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":843,"spinlock_wait_cycles":21888}
I20260620 14:28:18.605372 15501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7477c9325f0d465ba02ab41a0827dc8f" candidate_uuid: "003b0d6e179a459e8dbd975710b76112" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" is_pre_election: true
I20260620 14:28:18.605677 15501 raft_consensus.cc:2454] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 003b0d6e179a459e8dbd975710b76112 in term 1.
I20260620 14:28:18.606597 15299 leader_election.cc:302] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003b0d6e179a459e8dbd975710b76112, 5fbc961a9ac2442fa66df3a23bd39ae3; no voters:
W20260620 14:28:18.606794 15298 leader_election.cc:341] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733): Not found: Tablet not found: 7477c9325f0d465ba02ab41a0827dc8f
I20260620 14:28:18.606869 15687 raft_consensus.cc:2781] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:28:18.606998 15687 raft_consensus.cc:482] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:18.607057 15687 raft_consensus.cc:3037] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:28:18.609009 15687 raft_consensus.cc:504] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:18.609390 15687 leader_election.cc:288] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [CANDIDATE]: Term 2 election: Requested vote from peers d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733), 5fbc961a9ac2442fa66df3a23bd39ae3 (127.0.98.2:43667)
I20260620 14:28:18.609760 15501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7477c9325f0d465ba02ab41a0827dc8f" candidate_uuid: "003b0d6e179a459e8dbd975710b76112" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3"
I20260620 14:28:18.609905 15501 raft_consensus.cc:3037] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 1 FOLLOWER]: Advancing to term 2
W20260620 14:28:18.609979 15298 leader_election.cc:341] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733): Not found: Tablet not found: 7477c9325f0d465ba02ab41a0827dc8f
I20260620 14:28:18.611739 15501 raft_consensus.cc:2454] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 003b0d6e179a459e8dbd975710b76112 in term 2.
I20260620 14:28:18.612037 15299 leader_election.cc:302] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 003b0d6e179a459e8dbd975710b76112, 5fbc961a9ac2442fa66df3a23bd39ae3; no voters: d66e7e007aa64175ae7cdf8fc18d50f2
I20260620 14:28:18.612200 15687 raft_consensus.cc:2781] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:28:18.612378 15687 raft_consensus.cc:686] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [term 2 LEADER]: Becoming Leader. State: Replica: 003b0d6e179a459e8dbd975710b76112, State: Running, Role: LEADER
I20260620 14:28:18.612701 15687 consensus_queue.cc:237] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:18.613010 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:18.612772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:33199)
0620 14:28:18.612897 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:18.612947 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 14:28:19.048919 15501 raft_consensus.cc:1261] T 7477c9325f0d465ba02ab41a0827dc8f P 5fbc961a9ac2442fa66df3a23bd39ae3 [term 2 FOLLOWER]: Refusing update from remote peer 003b0d6e179a459e8dbd975710b76112: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 14:28:19.049638 15687 consensus_queue.cc:1035] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 14:28:19.051671 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:19.051292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:42539)
0620 14:28:19.051503 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:19.051559 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":50,"threads_started":1}
I20260620 14:28:19.053944 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:19.053659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:39973)
0620 14:28:19.053784 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:19.053843 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:28:19.054250 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:19.054045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:44329)
0620 14:28:19.054136 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:19.054188 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 14:28:19.054564 15298 consensus_peers.cc:487] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112 -> Peer d66e7e007aa64175ae7cdf8fc18d50f2 (127.0.98.3:37733): Couldn't send request to peer d66e7e007aa64175ae7cdf8fc18d50f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7477c9325f0d465ba02ab41a0827dc8f. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:28:19.462544 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:19.462209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:35481)
0620 14:28:19.462423 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:19.462470 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
I20260620 14:28:19.564934 15700 ts_tablet_manager.cc:927] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Initiating tablet copy from peer 003b0d6e179a459e8dbd975710b76112 (127.0.98.1:44607)
I20260620 14:28:19.565604 15700 tablet_copy_client.cc:316] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:44607
Jun 20 14:28:19 dist-test-slave-pk3j krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 14:28:19.581391 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:19.566139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44607 (local address 127.0.98.3:42601)
0620 14:28:19.566251 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:19.566277 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:19.566412 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:19.566987 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:19.566992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:19.567362 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:19.567473 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:19.567479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:19.568381 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:19.568384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:19.569028 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:19.569036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:19.569632 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:19.569635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:19.569715 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:19.576373 (+ 6658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:19.576385 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:19.577841 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:19.579822 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:19.579829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:19.579833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:19.580045 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:19.580283 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:19.580285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:19.580286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:19.580354 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:19.580736 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:19.580740 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:19.580873 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:28:19.580991 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:28:19.581181 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:28:19.581434 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:19.566301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42601 (local address 127.0.98.1:44607)
0620 14:28:19.566552 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:19.566560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:19.566587 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:28:19.566645 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:19.566649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:19.566828 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:19.566950 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:19.567576 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:19.568288 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:19.569125 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:19.569521 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:19.569605 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:19.577977 (+ 8372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:19.577985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:19.577997 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:19.578020 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:19.579730 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:19.580133 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:19.580138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:19.580139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:19.580210 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:19.580425 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:19.580427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:19.580428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:19.580646 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:19.580744 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:19.581085 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 14:28:19.581252 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
I20260620 14:28:19.582587 15385 tablet_copy_service.cc:140] P 003b0d6e179a459e8dbd975710b76112: Received BeginTabletCopySession request for tablet 7477c9325f0d465ba02ab41a0827dc8f from peer d66e7e007aa64175ae7cdf8fc18d50f2 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42601)
I20260620 14:28:19.582765 15385 tablet_copy_service.cc:161] P 003b0d6e179a459e8dbd975710b76112: Beginning new tablet copy session on tablet 7477c9325f0d465ba02ab41a0827dc8f from peer d66e7e007aa64175ae7cdf8fc18d50f2 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42601: session id = d66e7e007aa64175ae7cdf8fc18d50f2-7477c9325f0d465ba02ab41a0827dc8f
I20260620 14:28:19.583884 15385 tablet_copy_source_session.cc:205] T 7477c9325f0d465ba02ab41a0827dc8f P 003b0d6e179a459e8dbd975710b76112: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:28:19.587525 15700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7477c9325f0d465ba02ab41a0827dc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:19.593806 15700 tablet_copy_client.cc:791] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: tablet copy: Starting download of 0 data blocks...
I20260620 14:28:19.594007 15700 tablet_copy_client.cc:655] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: tablet copy: Starting download of 1 WAL segments...
I20260620 14:28:19.595477 15700 tablet_copy_client.cc:531] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:28:19.596845 15700 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Bootstrap starting.
I20260620 14:28:19.601756 15700 log.cc:824] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:19.607563 15700 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:19.607805 15700 tablet_bootstrap.cc:492] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Bootstrap complete.
I20260620 14:28:19.608036 15700 ts_tablet_manager.cc:1397] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260620 14:28:19.610121 15700 raft_consensus.cc:348] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:19.610325 15700 raft_consensus.cc:729] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66e7e007aa64175ae7cdf8fc18d50f2, State: Initialized, Role: FOLLOWER
I20260620 14:28:19.610672 15700 consensus_queue.cc:260] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d66e7e007aa64175ae7cdf8fc18d50f2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37733 } } peers { permanent_uuid: "5fbc961a9ac2442fa66df3a23bd39ae3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43667 } } peers { permanent_uuid: "003b0d6e179a459e8dbd975710b76112" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44607 } }
I20260620 14:28:19.611482 15700 ts_tablet_manager.cc:1428] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260620 14:28:19.611685 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:19.611446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:46445)
0620 14:28:19.611563 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:19.611623 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:28:19.612099 15385 tablet_copy_service.cc:342] P 003b0d6e179a459e8dbd975710b76112: Request end of tablet copy session d66e7e007aa64175ae7cdf8fc18d50f2-7477c9325f0d465ba02ab41a0827dc8f received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42601
I20260620 14:28:19.612201 15385 tablet_copy_service.cc:433] P 003b0d6e179a459e8dbd975710b76112: ending tablet copy session d66e7e007aa64175ae7cdf8fc18d50f2-7477c9325f0d465ba02ab41a0827dc8f on tablet 7477c9325f0d465ba02ab41a0827dc8f with peer d66e7e007aa64175ae7cdf8fc18d50f2
I20260620 14:28:19.612434 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:19.612197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:36849)
0620 14:28:19.612329 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:19.612377 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:28:19.990135 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:19.989755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:58891)
0620 14:28:19.990006 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:19.990047 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260620 14:28:19.990056 15634 raft_consensus.cc:1203] T 7477c9325f0d465ba02ab41a0827dc8f P d66e7e007aa64175ae7cdf8fc18d50f2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 14:28:19.990962 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:19.990708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:59031)
0620 14:28:19.990829 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:19.990882 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:28:20.055208 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:20.054869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:46739)
0620 14:28:20.055074 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:20.055121 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
I20260620 14:28:20.991124 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:20.990776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:33627)
0620 14:28:20.990988 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:20.991034 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
I20260620 14:28:20.991961 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:20.991587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:48193)
0620 14:28:20.991830 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:20.991886 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260620 14:28:21.056241 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:21.055853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:42515)
0620 14:28:21.056106 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:21.056156 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":66,"threads_started":1}
I20260620 14:28:21.992209 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:21.991838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:55241)
0620 14:28:21.992064 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:21.992115 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260620 14:28:21.992954 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:21.992604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:38209)
0620 14:28:21.992822 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:21.992869 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
I20260620 14:28:22.057430 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:22.056992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:51499)
0620 14:28:22.057282 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:22.057333 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":62,"threads_started":1}
I20260620 14:28:22.993484 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:22.993049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:60943)
0620 14:28:22.993329 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:22.993389 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":83,"threads_started":1}
I20260620 14:28:22.994145 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:22.993751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:53847)
0620 14:28:22.993989 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:22.994046 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
I20260620 14:28:22.994234 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:22.994000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:57589)
0620 14:28:22.994109 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:22.994153 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:28:23.058511 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:23.058158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:34453)
0620 14:28:23.058381 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:23.058429 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260620 14:28:23.995291 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:23.994838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:38583)
0620 14:28:23.995149 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:23.995202 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":87,"threads_started":1}
I20260620 14:28:23.995304 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:23.994879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:59977)
0620 14:28:23.995171 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:23.995222 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":64,"threads_started":1}
I20260620 14:28:24.059572 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:24.059220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:54811)
0620 14:28:24.059437 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:24.059483 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
I20260620 14:28:24.996519 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:24.996043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:47353)
0620 14:28:24.996348 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:24.996412 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
I20260620 14:28:24.996541 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:24.996076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:51853)
0620 14:28:24.996347 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:24.996396 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":58,"threads_started":1}
I20260620 14:28:25.060700 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:25.060253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:34781)
0620 14:28:25.060540 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:25.060601 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
I20260620 14:28:25.997852 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:25.997403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:36419)
0620 14:28:25.997655 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:25.997715 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":67,"threads_started":1}
I20260620 14:28:25.998083 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:25.997684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:58209)
0620 14:28:25.997911 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:25.997961 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260620 14:28:26.061815 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:26.061444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:34817)
0620 14:28:26.061662 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:26.061712 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
I20260620 14:28:26.999006 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:26.998606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:36645)
0620 14:28:26.998867 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:26.998917 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260620 14:28:26.999172 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:26.998760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:45921)
0620 14:28:26.999000 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:26.999048 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
I20260620 14:28:27.062963 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:27.062463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:58287)
0620 14:28:27.062825 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:27.062878 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":93,"threads_started":1}
I20260620 14:28:28.000214 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:27.999857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:55661)
0620 14:28:28.000076 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:28.000126 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260620 14:28:28.000264 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:27.999857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:38121)
0620 14:28:28.000138 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:28.000184 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":81,"threads_started":1}
I20260620 14:28:28.064100 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:28.063645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:44307)
0620 14:28:28.063959 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:28.064011 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":80,"threads_started":1}
I20260620 14:28:29.001264 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:29.000865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:51535)
0620 14:28:29.001085 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:29.001133 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260620 14:28:29.001332 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:29.000865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:47459)
0620 14:28:29.001146 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:29.001190 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":82,"threads_started":1}
I20260620 14:28:29.065101 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:29.064741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:55865)
0620 14:28:29.064960 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:29.065018 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
I20260620 14:28:30.002305 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:30.001885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:33479)
0620 14:28:30.002139 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:30.002199 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260620 14:28:30.002403 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:30.002027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:47089)
0620 14:28:30.002246 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:30.002296 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260620 14:28:30.066113 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:30.065771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:39441)
0620 14:28:30.065980 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:30.066028 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":61,"threads_started":1}
I20260620 14:28:31.003233 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:31.002876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:47287)
0620 14:28:31.003101 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:31.003146 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260620 14:28:31.003293 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:31.002973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:38781)
0620 14:28:31.003170 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:31.003215 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260620 14:28:31.067186 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:31.066814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:38929)
0620 14:28:31.067030 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:31.067080 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260620 14:28:32.004263 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:32.003894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.3:49569)
0620 14:28:32.004127 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:32.004176 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":88,"threads_started":1}
I20260620 14:28:32.004300 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:32.003894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.1:47375)
0620 14:28:32.004172 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:32.004221 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
I20260620 14:28:32.068192 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:32.067812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45975 (local address 127.0.98.2:42737)
0620 14:28:32.068051 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:32.068099 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260620 14:28:32.471063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 15247
I20260620 14:28:32.476961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 15415
I20260620 14:28:32.482138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 15551
2026-06-20T14:28:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19214 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:28:32 dist-test-slave-pk3j krb5kdc[15748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:28:32 dist-test-slave-pk3j krb5kdc[15748](info): set up 2 sockets
Jun 20 14:28:32 dist-test-slave-pk3j krb5kdc[15748](info): commencing operation
krb5kdc: starting...
W20260620 14:28:33.509646 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:28:33 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:28:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:28:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:28:33.620919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34885
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:33.727002 15761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:33.727308 15761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:33.727380 15761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:33.731642 15761 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:28:33.731702 15761 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:28:33.731725 15761 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:28:33.731776 15761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:33.731841 15761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:33.731881 15761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:33.731913 15761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:33.731953 15761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:33.737629 15761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:33.738566 15761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:33.739493 15761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:28:33.747804 15761 server_base.cc:1034] running on GCE node
W20260620 14:28:33.747783 15766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:33.747829 15769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:33.747822 15767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:33.748593 15761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:33.749699 15761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:33.750873 15761 hybrid_clock.cc:648] HybridClock initialized: now 1781965713750851 us; error 35 us; skew 500 ppm
Jun 20 14:28:33 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:33.753463 15761 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:33.754233 15761 webserver.cc:460] Webserver started at http://127.0.98.62:40545/ using document root <none> and password file <none>
I20260620 14:28:33.754731 15761 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:33.754786 15761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:33.754940 15761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:33.756798 15761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "032a22079ff945ddbe3a1d914408e56a"
format_stamp: "Formatted at 2026-06-20 14:28:33 on dist-test-slave-pk3j"
server_key: "3fbc15f9b3fd5ba31abd54b99b04b307"
server_key_iv: "93096cdadc4223b7414e5604158ad841"
server_key_version: "encryptionkey@0"
I20260620 14:28:33.757330 15761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "032a22079ff945ddbe3a1d914408e56a"
format_stamp: "Formatted at 2026-06-20 14:28:33 on dist-test-slave-pk3j"
server_key: "3fbc15f9b3fd5ba31abd54b99b04b307"
server_key_iv: "93096cdadc4223b7414e5604158ad841"
server_key_version: "encryptionkey@0"
I20260620 14:28:33.760702 15761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:28:33.762997 15776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:33.764091 15761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:28:33.764184 15761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "032a22079ff945ddbe3a1d914408e56a"
format_stamp: "Formatted at 2026-06-20 14:28:33 on dist-test-slave-pk3j"
server_key: "3fbc15f9b3fd5ba31abd54b99b04b307"
server_key_iv: "93096cdadc4223b7414e5604158ad841"
server_key_version: "encryptionkey@0"
I20260620 14:28:33.764264 15761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:33.779179 15761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:33.781796 15761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:33.781945 15761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:33.788528 15761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34885
I20260620 14:28:33.788797 15828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34885 every 8 connection(s)
I20260620 14:28:33.789492 15761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:33.792130 15829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:33.796211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 15761
I20260620 14:28:33.796339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:28:33.796588 392 external_mini_cluster.cc:1428] Setting key 15963fd399d7718930977e93b12e992d
I20260620 14:28:33.797704 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Bootstrap starting.
I20260620 14:28:33.799945 15829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:33.800776 15829 log.cc:824] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:33.802793 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: No bootstrap required, opened a new log
Jun 20 14:28:33 dist-test-slave-pk3j krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:33.807507 15829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:28:33.807763 15829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:33.807821 15829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a22079ff945ddbe3a1d914408e56a, State: Initialized, Role: FOLLOWER
I20260620 14:28:33.808308 15829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:28:33.808471 15829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:33.808537 15829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:33.808662 15829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:33.809572 15829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:28:33.809931 15829 leader_election.cc:302] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032a22079ff945ddbe3a1d914408e56a; no voters:
I20260620 14:28:33.810305 15829 leader_election.cc:288] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:28:33.810407 15834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:33.810647 15834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 1 LEADER]: Becoming Leader. State: Replica: 032a22079ff945ddbe3a1d914408e56a, State: Running, Role: LEADER
I20260620 14:28:33.810986 15834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:28:33.811188 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:33.797905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37050 (local address 127.0.98.62:34885)
0620 14:28:33.798299 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:33.798314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:33.798349 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:28:33.798952 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:33.798968 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:33.799274 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:33.799549 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:33.800461 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:33.801653 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:33.802747 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:33.803227 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:33.803340 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:33.806074 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:33.806082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:33.806103 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:33.806141 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:33.808107 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:33.808502 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:33.808508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:33.808513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:33.808697 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:33.809001 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:33.809007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:33.809010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:33.809441 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:33.809653 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:33.809929 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 14:28:33.810240 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":122,"threads_started":1}
I20260620 14:28:33.811556 15829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:33.812847 15835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "032a22079ff945ddbe3a1d914408e56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } } }
I20260620 14:28:33.812973 15835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:33.813359 15836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032a22079ff945ddbe3a1d914408e56a. Latest consensus state: current_term: 1 leader_uuid: "032a22079ff945ddbe3a1d914408e56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } } }
I20260620 14:28:33.813510 15836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:33.813939 15844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:33.816622 15844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:33.822233 15844 catalog_manager.cc:1347] Generated new cluster ID: 2d939793587e48f98b92e559b99abfed
I20260620 14:28:33.822293 15844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:33.848088 15844 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:28:33.849411 15844 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:33.860730 15844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Generated new TSK 0
I20260620 14:28:33.861570 15844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:28:33.895397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34885
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:34.011389 15855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:34.011616 15855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:34.011684 15855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:34.016234 15855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:34.016311 15855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:34.016389 15855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:34.021447 15855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:34.022360 15855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:34.023329 15855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:34.032418 15860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:34.032483 15861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:34.032418 15863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:34.032604 15855 server_base.cc:1034] running on GCE node
I20260620 14:28:34.033037 15855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:34.033694 15855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:34.034865 15855 hybrid_clock.cc:648] HybridClock initialized: now 1781965714034844 us; error 32 us; skew 500 ppm
Jun 20 14:28:34 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:34.037518 15855 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:34.038316 15855 webserver.cc:460] Webserver started at http://127.0.98.1:44171/ using document root <none> and password file <none>
I20260620 14:28:34.038846 15855 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:34.038944 15855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:34.039152 15855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:34.041101 15855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "c5e3e035c975489da1933d543915e20b"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "71903ddc054320950128ef403ecf996c"
server_key_iv: "ce9ade81a818f14c792ab86119537783"
server_key_version: "encryptionkey@0"
I20260620 14:28:34.041684 15855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c5e3e035c975489da1933d543915e20b"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "71903ddc054320950128ef403ecf996c"
server_key_iv: "ce9ade81a818f14c792ab86119537783"
server_key_version: "encryptionkey@0"
I20260620 14:28:34.045017 15855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:34.047286 15870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.048411 15855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:34.048533 15855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "c5e3e035c975489da1933d543915e20b"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "71903ddc054320950128ef403ecf996c"
server_key_iv: "ce9ade81a818f14c792ab86119537783"
server_key_version: "encryptionkey@0"
I20260620 14:28:34.048635 15855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:34.058177 15855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:34.061126 15855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:34.061328 15855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:34.061861 15855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:34.062585 15855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:34.062649 15855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.062713 15855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:34.062758 15855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.071506 15855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45659
I20260620 14:28:34.071544 15983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45659 every 8 connection(s)
I20260620 14:28:34.072326 15855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:28:34 dist-test-slave-pk3j krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:34.080786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 15855
I20260620 14:28:34.080921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:28:34.081236 392 external_mini_cluster.cc:1428] Setting key 5bba17f62f690abf2b02c56a14e5b346
I20260620 14:28:34.085287 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.073946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58041 (local address 127.0.98.62:34885)
0620 14:28:34.074077 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.074082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.074868 (+ 786us) server_negotiation.cc:407] Connection header received
0620 14:28:34.076018 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.076021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.076066 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.076138 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:34.077147 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.077725 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.078766 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.078984 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.079057 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.081454 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:34.081457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:34.081459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:34.081481 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:34.083360 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.083843 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.083846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.083848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.083922 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.084241 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.084244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.084245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.084407 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:34.084507 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.084973 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.085098 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:28:34.086074 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.074338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:58041)
0620 14:28:34.074732 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.074771 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:34.075840 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:34.076279 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:34.076285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:34.076599 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:34.077034 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.077046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.077829 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.077832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.078666 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.078674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.079097 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.079100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.079230 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.079794 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:34.079809 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:34.081277 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:34.083483 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:34.083488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:34.083501 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:34.083737 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:34.084007 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:34.084011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:34.084013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:34.084144 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:34.084506 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:34.084512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:34.084729 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 14:28:34.084920 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:28:34.085215 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":107,"threads_started":1}
I20260620 14:28:34.087272 15984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34885
I20260620 14:28:34.087481 15984 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:34.087930 15984 heartbeater.cc:511] Master 127.0.98.62:34885 requested a full tablet report, sending...
I20260620 14:28:34.089390 15793 ts_manager.cc:194] Registered new tserver with Master: c5e3e035c975489da1933d543915e20b (127.0.98.1:45659)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:28:34.090502 15793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:28:34.104786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34885
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:34.208977 15989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:34.209236 15989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:34.209293 15989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:34.213263 15989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:34.213307 15989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:34.213362 15989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:34.218300 15989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:34.219125 15989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:34.219993 15989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:34.228693 15995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:34.228686 15994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:34.228855 15989 server_base.cc:1034] running on GCE node
W20260620 14:28:34.228686 15997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:34.229426 15989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:34.230070 15989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:34.231240 15989 hybrid_clock.cc:648] HybridClock initialized: now 1781965714231231 us; error 30 us; skew 500 ppm
Jun 20 14:28:34 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:34.234133 15989 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:34.234836 15989 webserver.cc:460] Webserver started at http://127.0.98.2:33495/ using document root <none> and password file <none>
I20260620 14:28:34.235316 15989 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:34.235374 15989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:34.235519 15989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:34.237376 15989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "1172c6ed388143669b766dcb6caa8e97"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "2374ec441f0763a15e069fe5a5eb979e"
server_key_iv: "fd35bf7a515aa3988ba9c5fc1c9d66e1"
server_key_version: "encryptionkey@0"
I20260620 14:28:34.237877 15989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1172c6ed388143669b766dcb6caa8e97"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "2374ec441f0763a15e069fe5a5eb979e"
server_key_iv: "fd35bf7a515aa3988ba9c5fc1c9d66e1"
server_key_version: "encryptionkey@0"
I20260620 14:28:34.241283 15989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:28:34.243376 16004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.244484 15989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:28:34.244572 15989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "1172c6ed388143669b766dcb6caa8e97"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "2374ec441f0763a15e069fe5a5eb979e"
server_key_iv: "fd35bf7a515aa3988ba9c5fc1c9d66e1"
server_key_version: "encryptionkey@0"
I20260620 14:28:34.244647 15989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:34.256426 15989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:34.259523 15989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:34.259666 15989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:34.260157 15989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:34.260908 15989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:34.260953 15989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.260990 15989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:34.261011 15989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.269536 15989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36451
I20260620 14:28:34.269845 16117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36451 every 8 connection(s)
I20260620 14:28:34.270442 15989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:28:34 dist-test-slave-pk3j krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:34.279903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 15989
I20260620 14:28:34.280041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:28:34.280367 392 external_mini_cluster.cc:1428] Setting key 095ec66e352d498b742cb5cf8fc1bdb4
I20260620 14:28:34.283097 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.272148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33099 (local address 127.0.98.62:34885)
0620 14:28:34.272272 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.272276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.273069 (+ 793us) server_negotiation.cc:407] Connection header received
0620 14:28:34.273930 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.273933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.273980 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.274068 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:34.275095 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.275616 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.276615 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.276836 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.276907 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.279263 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:34.279266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:34.279268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:34.279290 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:34.280855 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.281424 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.281426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.281427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.281503 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.281892 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.281895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.281897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.282125 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:34.282223 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.282700 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.282846 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:28:34.283666 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.272520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:33099)
0620 14:28:34.272929 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.272969 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:34.273736 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:34.274192 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:34.274198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:34.274528 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:34.274981 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.274993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.275711 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.275713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.276523 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.276531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.276953 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.276956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.277090 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.277730 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:34.277745 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:34.279125 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:34.280960 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:34.280966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:34.280982 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:34.281336 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:34.281590 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:34.281593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:34.281595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:34.281772 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:34.282232 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:34.282237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:34.282458 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 14:28:34.282599 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 14:28:34.282846 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":115,"threads_started":1}
I20260620 14:28:34.285048 16118 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34885
I20260620 14:28:34.285332 16118 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:34.285784 16118 heartbeater.cc:511] Master 127.0.98.62:34885 requested a full tablet report, sending...
I20260620 14:28:34.286791 15793 ts_manager.cc:194] Registered new tserver with Master: 1172c6ed388143669b766dcb6caa8e97 (127.0.98.2:36451)
I20260620 14:28:34.287333 15793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33099
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:28:34.304766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34885
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:34.410352 16123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:34.410630 16123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:34.410696 16123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:34.414844 16123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:34.414922 16123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:34.415009 16123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:34.420147 16123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:34.421180 16123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:34.422204 16123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:34.431212 16128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:34.431212 16131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:34.431401 16123 server_base.cc:1034] running on GCE node
W20260620 14:28:34.431289 16129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:34.431902 16123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:34.432453 16123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:34.433622 16123 hybrid_clock.cc:648] HybridClock initialized: now 1781965714433603 us; error 31 us; skew 500 ppm
Jun 20 14:28:34 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:34.436293 16123 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:34.437110 16123 webserver.cc:460] Webserver started at http://127.0.98.3:38231/ using document root <none> and password file <none>
I20260620 14:28:34.437655 16123 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:34.437745 16123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:34.437955 16123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:28:34.439891 16123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "38d73562a73a4e3db87b6586ad4bb868"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "f3ad35002fe062b04b755490589c6b18"
server_key_iv: "0af4047ffec98c5751971a04adfc22ae"
server_key_version: "encryptionkey@0"
I20260620 14:28:34.440419 16123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "38d73562a73a4e3db87b6586ad4bb868"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "f3ad35002fe062b04b755490589c6b18"
server_key_iv: "0af4047ffec98c5751971a04adfc22ae"
server_key_version: "encryptionkey@0"
I20260620 14:28:34.443961 16123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:28:34.446197 16138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.447329 16123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:28:34.447455 16123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "38d73562a73a4e3db87b6586ad4bb868"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "f3ad35002fe062b04b755490589c6b18"
server_key_iv: "0af4047ffec98c5751971a04adfc22ae"
server_key_version: "encryptionkey@0"
I20260620 14:28:34.447552 16123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:34.472263 16123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:34.474994 16123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:34.475180 16123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:34.475729 16123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:34.476491 16123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:34.476557 16123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.476622 16123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:34.476661 16123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:34.484841 16123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38301
I20260620 14:28:34.484870 16251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38301 every 8 connection(s)
I20260620 14:28:34.485673 16123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:28:34.490551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 16123
I20260620 14:28:34.490677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:28:34.490967 392 external_mini_cluster.cc:1428] Setting key d9871f2a05ca489a615f7eba72b64132
Jun 20 14:28:34 dist-test-slave-pk3j krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:34.498824 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.487360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40225 (local address 127.0.98.62:34885)
0620 14:28:34.487530 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.487535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.488299 (+ 764us) server_negotiation.cc:407] Connection header received
0620 14:28:34.489602 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.489605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.489670 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.489735 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:34.490931 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.491600 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.493171 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.493443 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.493522 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.495667 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:34.495670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:34.495672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:34.495693 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:34.497102 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.497578 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.497580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.497581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.497652 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.497937 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.497939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.497940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.498094 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:34.498163 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.498603 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.498694 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:28:34.499619 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.487730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:40225)
0620 14:28:34.488159 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.488201 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:34.489406 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:34.489888 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:34.489894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:34.490244 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:34.490783 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.490803 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.491719 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.491726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.493073 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.493083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.493564 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.493567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.493705 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.494223 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:34.494239 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:34.495534 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:34.497191 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:34.497195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:34.497245 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:34.497495 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:34.497730 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:34.497732 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:34.497733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:34.497866 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:34.498172 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:34.498177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:34.498414 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 14:28:34.498565 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:28:34.498796 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":113,"threads_started":1}
I20260620 14:28:34.500679 16252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34885
I20260620 14:28:34.500913 16252 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:34.501423 16252 heartbeater.cc:511] Master 127.0.98.62:34885 requested a full tablet report, sending...
I20260620 14:28:34.502373 15793 ts_manager.cc:194] Registered new tserver with Master: 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301)
I20260620 14:28:34.502907 15793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40225
I20260620 14:28:34.504510 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:28:34.513154 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.506365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.0.98.62:34885)
0620 14:28:34.506489 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.506493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.506618 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:28:34.506699 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.506701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.506745 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.506805 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:34.507421 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.507903 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.508651 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.508856 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.508927 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.510141 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:34.510144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:34.510146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:34.510166 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:34.511612 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.512012 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.512015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.512016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.512084 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:34.512343 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:34.512348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:34.512349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:34.512559 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:34.512657 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.512877 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.512963 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:28:34.515662 15793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:28:34.518715 15793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:28:34.532527 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.526750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36451 (local address 127.0.0.1:55196)
0620 14:28:34.527830 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.527841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:34.527940 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:34.528429 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:34.528435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:34.528460 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:34.528599 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.528606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.529807 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.529811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.531091 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.531101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.531989 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.531992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.532104 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.532142 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:28:34.532262 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:28:34.532377 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":96,"threads_started":1}
I20260620 14:28:34.532557 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.527402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.0.98.2:36451)
0620 14:28:34.527823 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.527836 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.527927 (+ 91us) server_negotiation.cc:407] Connection header received
0620 14:28:34.528052 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.528056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.528268 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.528418 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:34.528733 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.529680 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.531217 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.531872 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.531977 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.532114 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.532299 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.532394 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":128,"threads_started":1}
I20260620 14:28:34.533159 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.526958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45659 (local address 127.0.0.1:60404)
0620 14:28:34.528097 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.528108 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:34.528206 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:34.528605 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:34.528608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:34.528625 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:34.528723 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.528730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.530038 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.530041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.531517 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.531527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.532775 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.532778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.532854 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.532871 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:28:34.532954 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:28:34.533029 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1092,"mutex_wait_us":32,"thread_start_us":83,"threads_started":1}
I20260620 14:28:34.533655 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.526569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38301 (local address 127.0.0.1:39396)
0620 14:28:34.527307 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.527356 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:34.527483 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:34.529012 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:34.529015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:34.529032 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:34.529109 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.529115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.530502 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.530505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.531657 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.531664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.532890 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.532894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.532988 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.533013 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:28:34.533083 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:28:34.533161 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":17,"thread_start_us":86,"threads_started":1}
I20260620 14:28:34.533919 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.527397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.0.98.1:45659)
0620 14:28:34.527992 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.528001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.528220 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:28:34.528305 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.528310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.528488 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.528630 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:34.528827 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.529889 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.531652 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.532452 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.532560 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.532712 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.533002 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.533699 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":141,"threads_started":1}
I20260620 14:28:34.534003 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.526721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.0.98.3:38301)
0620 14:28:34.528434 (+ 1713us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.528445 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.528465 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:28:34.528531 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.528536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.528724 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.528861 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:34.529257 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.530349 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.531784 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.532574 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.532698 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.532833 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.533269 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.533364 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1600,"thread_start_us":134,"threads_started":1}
I20260620 14:28:34.536124 16186 tablet_service.cc:1467] Processing CreateTablet for tablet e8e85c2ba1ac420dae3d36945fab7da3 (DEFAULT_TABLE table=test-table [id=41922daf8b3c40248702b41ddf905086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:34.536145 16052 tablet_service.cc:1467] Processing CreateTablet for tablet e8e85c2ba1ac420dae3d36945fab7da3 (DEFAULT_TABLE table=test-table [id=41922daf8b3c40248702b41ddf905086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:34.536381 15918 tablet_service.cc:1467] Processing CreateTablet for tablet e8e85c2ba1ac420dae3d36945fab7da3 (DEFAULT_TABLE table=test-table [id=41922daf8b3c40248702b41ddf905086]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:34.537110 16186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e85c2ba1ac420dae3d36945fab7da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:34.537156 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e85c2ba1ac420dae3d36945fab7da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:34.537184 15918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8e85c2ba1ac420dae3d36945fab7da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:34.542922 16269 tablet_bootstrap.cc:492] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: Bootstrap starting.
I20260620 14:28:34.544088 16271 tablet_bootstrap.cc:492] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Bootstrap starting.
I20260620 14:28:34.544492 16270 tablet_bootstrap.cc:492] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: Bootstrap starting.
I20260620 14:28:34.545959 16269 tablet_bootstrap.cc:654] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:34.546967 16271 tablet_bootstrap.cc:654] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:34.547019 16270 tablet_bootstrap.cc:654] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:34.547780 16269 log.cc:824] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:34.547860 16270 log.cc:824] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:34.548101 16271 log.cc:824] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:34.549752 16269 tablet_bootstrap.cc:492] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: No bootstrap required, opened a new log
I20260620 14:28:34.549842 16271 tablet_bootstrap.cc:492] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: No bootstrap required, opened a new log
I20260620 14:28:34.549849 16270 tablet_bootstrap.cc:492] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: No bootstrap required, opened a new log
I20260620 14:28:34.549981 16269 ts_tablet_manager.cc:1397] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:28:34.550026 16271 ts_tablet_manager.cc:1397] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:28:34.550026 16270 ts_tablet_manager.cc:1397] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 14:28:34.553642 16269 raft_consensus.cc:348] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:34.553889 16269 raft_consensus.cc:374] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:34.553967 16269 raft_consensus.cc:729] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d73562a73a4e3db87b6586ad4bb868, State: Initialized, Role: FOLLOWER
I20260620 14:28:34.553828 16270 raft_consensus.cc:348] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:34.553920 16271 raft_consensus.cc:348] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:34.554117 16271 raft_consensus.cc:374] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:34.554132 16270 raft_consensus.cc:374] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:34.554183 16271 raft_consensus.cc:729] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e3e035c975489da1933d543915e20b, State: Initialized, Role: FOLLOWER
I20260620 14:28:34.554189 16270 raft_consensus.cc:729] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1172c6ed388143669b766dcb6caa8e97, State: Initialized, Role: FOLLOWER
I20260620 14:28:34.554486 16269 consensus_queue.cc:260] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:34.554605 16270 consensus_queue.cc:260] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:34.554605 16271 consensus_queue.cc:260] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:34.555426 16118 heartbeater.cc:503] Master 127.0.98.62:34885 was elected leader, sending a full tablet report...
I20260620 14:28:34.555544 16269 ts_tablet_manager.cc:1428] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:28:34.555546 16271 ts_tablet_manager.cc:1428] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:28:34.555546 16270 ts_tablet_manager.cc:1428] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260620 14:28:34.555836 16252 heartbeater.cc:503] Master 127.0.98.62:34885 was elected leader, sending a full tablet report...
I20260620 14:28:34.556584 15984 heartbeater.cc:503] Master 127.0.98.62:34885 was elected leader, sending a full tablet report...
I20260620 14:28:34.569810 16277 raft_consensus.cc:482] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:34.570019 16277 raft_consensus.cc:504] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:34.571156 16277 leader_election.cc:288] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301), c5e3e035c975489da1933d543915e20b (127.0.98.1:45659)
W20260620 14:28:34.573398 15985 tablet.cc:2369] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:34.576025 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.571524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41605 (local address 127.0.98.1:45659)
0620 14:28:34.571664 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.571669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.571914 (+ 245us) server_negotiation.cc:407] Connection header received
0620 14:28:34.572021 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.572023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.572062 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.572135 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:34.572345 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.573061 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.574718 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.575352 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.575461 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.575524 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.575833 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.575901 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:28:34.576025 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.571446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58825 (local address 127.0.98.3:38301)
0620 14:28:34.571585 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.571590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.571604 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:28:34.571754 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.571756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.571800 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.571863 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:34.572176 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.572958 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.574724 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.575352 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.575461 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.575524 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.575833 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.575901 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:28:34.576071 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.571331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38301 (local address 127.0.98.2:58825)
0620 14:28:34.571511 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.571528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:34.571618 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:34.571901 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:34.571904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:34.571929 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:34.572060 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.572065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.573061 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.573064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.574587 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.574598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.575471 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.575474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.575635 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.575728 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:28:34.575809 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:28:34.575901 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":188,"spinlock_wait_cycles":3712}
I20260620 14:28:34.576079 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.571407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45659 (local address 127.0.98.2:41605)
0620 14:28:34.571795 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.571806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:34.571927 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:34.572149 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:34.572152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:34.572171 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:34.572257 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.572261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.573146 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.573148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.574591 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:34.574598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.575471 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.575474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:34.575598 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.575728 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:28:34.575809 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:28:34.575901 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":178,"spinlock_wait_cycles":1408,"thread_start_us":134,"threads_started":1}
I20260620 14:28:34.576678 16206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e85c2ba1ac420dae3d36945fab7da3" candidate_uuid: "1172c6ed388143669b766dcb6caa8e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d73562a73a4e3db87b6586ad4bb868" is_pre_election: true
I20260620 14:28:34.576980 16206 raft_consensus.cc:2454] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1172c6ed388143669b766dcb6caa8e97 in term 0.
I20260620 14:28:34.576956 15938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e85c2ba1ac420dae3d36945fab7da3" candidate_uuid: "1172c6ed388143669b766dcb6caa8e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e3e035c975489da1933d543915e20b" is_pre_election: true
I20260620 14:28:34.577239 15938 raft_consensus.cc:2454] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1172c6ed388143669b766dcb6caa8e97 in term 0.
I20260620 14:28:34.577461 16007 leader_election.cc:302] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1172c6ed388143669b766dcb6caa8e97, 38d73562a73a4e3db87b6586ad4bb868; no voters:
I20260620 14:28:34.577723 16277 raft_consensus.cc:2781] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:28:34.577806 16277 raft_consensus.cc:482] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:34.577863 16277 raft_consensus.cc:3037] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:34.578967 16277 raft_consensus.cc:504] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:34.579290 16277 leader_election.cc:288] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 1 election: Requested vote from peers 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301), c5e3e035c975489da1933d543915e20b (127.0.98.1:45659)
I20260620 14:28:34.579577 16206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e85c2ba1ac420dae3d36945fab7da3" candidate_uuid: "1172c6ed388143669b766dcb6caa8e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d73562a73a4e3db87b6586ad4bb868"
I20260620 14:28:34.579663 15938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8e85c2ba1ac420dae3d36945fab7da3" candidate_uuid: "1172c6ed388143669b766dcb6caa8e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e3e035c975489da1933d543915e20b"
I20260620 14:28:34.579704 16206 raft_consensus.cc:3037] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:34.579810 15938 raft_consensus.cc:3037] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:34.580703 16206 raft_consensus.cc:2454] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1172c6ed388143669b766dcb6caa8e97 in term 1.
I20260620 14:28:34.580829 15938 raft_consensus.cc:2454] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1172c6ed388143669b766dcb6caa8e97 in term 1.
I20260620 14:28:34.581009 16007 leader_election.cc:302] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1172c6ed388143669b766dcb6caa8e97, 38d73562a73a4e3db87b6586ad4bb868; no voters:
I20260620 14:28:34.581174 16277 raft_consensus.cc:2781] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:34.581406 16277 raft_consensus.cc:686] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 LEADER]: Becoming Leader. State: Replica: 1172c6ed388143669b766dcb6caa8e97, State: Running, Role: LEADER
I20260620 14:28:34.581746 16277 consensus_queue.cc:237] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:34.584722 15790 catalog_manager.cc:5495] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1172c6ed388143669b766dcb6caa8e97 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1172c6ed388143669b766dcb6caa8e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } health_report { overall_health: UNKNOWN } } }
I20260620 14:28:34.599341 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.595169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.0.98.2:36451)
0620 14:28:34.595299 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:34.595303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:34.595326 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:28:34.595492 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:34.595494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:34.595545 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:34.595649 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:34.596043 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.596529 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.597926 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:34.598136 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:34.598293 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:34.598580 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:34.598980 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:28:34.599105 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:34.599148 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:28:34.599201 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:28:34.604795 16206 raft_consensus.cc:1261] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868 [term 1 FOLLOWER]: Refusing update from remote peer 1172c6ed388143669b766dcb6caa8e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:34.604930 15938 raft_consensus.cc:1261] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Refusing update from remote peer 1172c6ed388143669b766dcb6caa8e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:34.605577 16277 consensus_queue.cc:1035] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:34.605762 16279 consensus_queue.cc:1035] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:34.617559 16285 mvcc.cc:205] Tried to move back new op lower bound from 7298931567015030784 to 7298931566929604608. Current Snapshot: MvccSnapshot[applied={T|T < 7298931567015030784}]
I20260620 14:28:34.619702 15790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:34.619869 15790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:34.622268 15790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Sending DeleteTablet for 3 replicas of tablet e8e85c2ba1ac420dae3d36945fab7da3
I20260620 14:28:34.622964 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet e8e85c2ba1ac420dae3d36945fab7da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55196
I20260620 14:28:34.623036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 15855
I20260620 14:28:34.623169 16186 tablet_service.cc:1514] Processing DeleteTablet for tablet e8e85c2ba1ac420dae3d36945fab7da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39396
I20260620 14:28:34.623804 16293 tablet_replica.cc:330] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: stopping tablet replica
I20260620 14:28:34.624150 16293 raft_consensus.cc:2229] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:34.624439 16293 raft_consensus.cc:2258] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:34.624847 16292 tablet_replica.cc:330] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: stopping tablet replica
I20260620 14:28:34.625181 16292 raft_consensus.cc:2229] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:28:34.625850 16292 raft_consensus.cc:2258] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:34.626021 16293 ts_tablet_manager.cc:1905] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:34.627221 16292 ts_tablet_manager.cc:1905] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:34.628731 16293 ts_tablet_manager.cc:1918] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:34.628886 16293 log.cc:1196] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/e8e85c2ba1ac420dae3d36945fab7da3
I20260620 14:28:34.629380 16293 ts_tablet_manager.cc:1939] T e8e85c2ba1ac420dae3d36945fab7da3 P 38d73562a73a4e3db87b6586ad4bb868: Deleting consensus metadata
I20260620 14:28:34.630728 16292 ts_tablet_manager.cc:1918] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:34.630832 16292 log.cc:1196] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/e8e85c2ba1ac420dae3d36945fab7da3
I20260620 14:28:34.631194 16292 ts_tablet_manager.cc:1939] T e8e85c2ba1ac420dae3d36945fab7da3 P 1172c6ed388143669b766dcb6caa8e97: Deleting consensus metadata
I20260620 14:28:34.631201 15779 catalog_manager.cc:4841] TS 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301): tablet e8e85c2ba1ac420dae3d36945fab7da3 (table test-table [id=41922daf8b3c40248702b41ddf905086]) successfully deleted
I20260620 14:28:34.632335 15780 catalog_manager.cc:4841] TS 1172c6ed388143669b766dcb6caa8e97 (127.0.98.2:36451): tablet e8e85c2ba1ac420dae3d36945fab7da3 (table test-table [id=41922daf8b3c40248702b41ddf905086]) successfully deleted
W20260620 14:28:34.632896 15778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45659 (error 108)
I20260620 14:28:34.633080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 15989
I20260620 14:28:34.638082 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:34.637740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45659 (local address 127.0.0.1:60416)
0620 14:28:34.637894 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:34.637976 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 14:28:34.638250 15778 catalog_manager.cc:4584] TS c5e3e035c975489da1933d543915e20b (127.0.98.1:45659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8e85c2ba1ac420dae3d36945fab7da3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45659: connect: Connection refused (error 111)
I20260620 14:28:34.639488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 16123
I20260620 14:28:34.645174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 15761
I20260620 14:28:54.651669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34885
--webserver_interface=127.0.98.62
--webserver_port=40545
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:28:54.756116 16295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:54.756419 16295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:54.756491 16295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:54.760505 16295 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:28:54.760596 16295 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:28:54.760624 16295 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:28:54.760654 16295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:28:54.760697 16295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:54.760735 16295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:54.760792 16295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:28:54.760831 16295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:28:54.765938 16295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:54.766965 16295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:54.767941 16295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:54.780248 16304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:54.780226 16302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:54.780226 16301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:54.780576 16295 server_base.cc:1034] running on GCE node
I20260620 14:28:54.781162 16295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:54.782289 16295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:54.783454 16295 hybrid_clock.cc:648] HybridClock initialized: now 1781965734783439 us; error 28 us; skew 500 ppm
Jun 20 14:28:54 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:54.792306 16295 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:28:54.793290 16295 webserver.cc:460] Webserver started at http://127.0.98.62:40545/ using document root <none> and password file <none>
I20260620 14:28:54.793880 16295 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:54.793953 16295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:54.797835 16295 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:54.799823 16311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:54.800999 16295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:28:54.801105 16295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "032a22079ff945ddbe3a1d914408e56a"
format_stamp: "Formatted at 2026-06-20 14:28:33 on dist-test-slave-pk3j"
server_key: "3fbc15f9b3fd5ba31abd54b99b04b307"
server_key_iv: "93096cdadc4223b7414e5604158ad841"
server_key_version: "encryptionkey@0"
I20260620 14:28:54.801743 16295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:54.815395 16295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:54.817989 16295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:54.818156 16295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:54.824946 16295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34885
I20260620 14:28:54.825287 16363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34885 every 8 connection(s)
I20260620 14:28:54.826072 16295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:28:54.827569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 16295
I20260620 14:28:54.827953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45659
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34885
--webserver_port=44171
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:54.830447 16364 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:28:54.832103 16364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Bootstrap starting.
I20260620 14:28:54.855055 16364 log.cc:824] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:54.866708 16364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:54.867249 16364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Bootstrap complete.
I20260620 14:28:54.874131 16364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:28:54.874748 16364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a22079ff945ddbe3a1d914408e56a, State: Initialized, Role: FOLLOWER
I20260620 14:28:54.875345 16364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:28:54.875516 16364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:28:54.875649 16364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:28:54.875771 16364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:28:54.878639 16364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:28:54.879024 16364 leader_election.cc:302] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032a22079ff945ddbe3a1d914408e56a; no voters:
I20260620 14:28:54.879410 16364 leader_election.cc:288] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:28:54.879531 16368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:28:54.879827 16368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 2 LEADER]: Becoming Leader. State: Replica: 032a22079ff945ddbe3a1d914408e56a, State: Running, Role: LEADER
I20260620 14:28:54.880188 16368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:28:54.880846 16364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:28:54.881143 16368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032a22079ff945ddbe3a1d914408e56a. Latest consensus state: current_term: 2 leader_uuid: "032a22079ff945ddbe3a1d914408e56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } } }
I20260620 14:28:54.881974 16368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:54.881822 16369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "032a22079ff945ddbe3a1d914408e56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } } }
I20260620 14:28:54.882764 16369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: This master's current role is: LEADER
I20260620 14:28:54.882992 16377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:28:54.887518 16377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:28:54.888123 16377 catalog_manager.cc:1259] Loaded cluster ID: 2d939793587e48f98b92e559b99abfed
I20260620 14:28:54.888185 16377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:28:54.890555 16377 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:28:54.891273 16377 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Loaded TSK: 0
I20260620 14:28:54.908890 16377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Generated new TSK 1
I20260620 14:28:54.909134 16377 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:28:54.952204 16366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:54.952483 16366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:54.952572 16366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:54.956565 16366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:54.956651 16366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:54.956739 16366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:28:54.961782 16366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44171
--tserver_master_addrs=127.0.98.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:54.962788 16366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:54.963822 16366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:54.974000 16394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:54.974010 16391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:54.974053 16392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:54.974512 16366 server_base.cc:1034] running on GCE node
I20260620 14:28:54.974900 16366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:54.975504 16366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:54.976722 16366 hybrid_clock.cc:648] HybridClock initialized: now 1781965734976696 us; error 40 us; skew 500 ppm
Jun 20 14:28:54 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:54.979346 16366 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:28:54.980266 16366 webserver.cc:460] Webserver started at http://127.0.98.1:44171/ using document root <none> and password file <none>
I20260620 14:28:54.980789 16366 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:54.980859 16366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:54.984805 16366 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:28:54.986871 16401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:54.988070 16366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:28:54.988158 16366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "c5e3e035c975489da1933d543915e20b"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "71903ddc054320950128ef403ecf996c"
server_key_iv: "ce9ade81a818f14c792ab86119537783"
server_key_version: "encryptionkey@0"
I20260620 14:28:54.988677 16366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:55.003849 16366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:55.007191 16366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:55.007373 16366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:55.007961 16366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:55.009060 16408 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:28:55.012948 16366 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:28:55.013020 16366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 14:28:55.013092 16366 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:28:55.014843 16366 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:28:55.014902 16366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 14:28:55.015159 16408 tablet_bootstrap.cc:492] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Bootstrap starting.
I20260620 14:28:55.025374 16366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45659
I20260620 14:28:55.025409 16515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45659 every 8 connection(s)
I20260620 14:28:55.026623 16366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:28:55.034250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 16366
I20260620 14:28:55.034690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36451
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34885
--webserver_port=33495
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:28:55.038100 16408 log.cc:824] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:28:55 dist-test-slave-pk3j krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:55.045153 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.028940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42445 (local address 127.0.98.62:34885)
0620 14:28:55.029355 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.029374 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.029984 (+ 610us) server_negotiation.cc:407] Connection header received
0620 14:28:55.031455 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.031473 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.031876 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.032236 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:55.033246 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.034833 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.036640 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.037284 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.037468 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.040383 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:55.040390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:55.040417 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:55.040442 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:55.042133 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:55.042642 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:55.042648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:55.042660 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:55.042768 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:55.043109 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:55.043114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:55.043115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:55.043438 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:55.043573 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.044084 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.044323 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":122,"threads_started":1}
I20260620 14:28:55.045171 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.028940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:42445)
0620 14:28:55.029787 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.029862 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:55.030915 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:55.032288 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:55.032295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:55.032621 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:55.033049 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.033062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.035009 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.035013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.036499 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.036510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.037649 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.037653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.037839 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.038538 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:55.038558 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:55.040192 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:55.042255 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:55.042261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:55.042274 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:55.042519 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:55.042873 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:55.042876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:55.042878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:55.043014 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:55.043562 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:55.043568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:55.043792 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 14:28:55.043965 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 14:28:55.044280 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":120,"threads_started":1}
I20260620 14:28:55.047636 16516 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34885
I20260620 14:28:55.047856 16516 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:55.048000 16408 tablet_bootstrap.cc:492] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:28:55.048350 16516 heartbeater.cc:511] Master 127.0.98.62:34885 requested a full tablet report, sending...
I20260620 14:28:55.048370 16408 tablet_bootstrap.cc:492] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Bootstrap complete.
I20260620 14:28:55.049806 16408 ts_tablet_manager.cc:1397] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.008s
I20260620 14:28:55.050683 16328 ts_manager.cc:194] Registered new tserver with Master: c5e3e035c975489da1933d543915e20b (127.0.98.1:45659)
I20260620 14:28:55.052620 16408 raft_consensus.cc:348] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.053144 16408 raft_consensus.cc:729] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e3e035c975489da1933d543915e20b, State: Initialized, Role: FOLLOWER
I20260620 14:28:55.053686 16408 consensus_queue.cc:260] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.054001 16328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42445
I20260620 14:28:55.054657 16408 ts_tablet_manager.cc:1428] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 14:28:55.056524 16516 heartbeater.cc:503] Master 127.0.98.62:34885 was elected leader, sending a full tablet report...
I20260620 14:28:55.059584 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.053404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.0.98.1:45659)
0620 14:28:55.055076 (+ 1672us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.055087 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.055106 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:28:55.055165 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.055169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.055678 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.055828 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:55.056154 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.057104 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.058384 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.058958 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.059085 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.059205 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.059383 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.059462 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1588,"thread_start_us":110,"threads_started":1}
I20260620 14:28:55.059597 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.053760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45659 (local address 127.0.0.1:34094)
0620 14:28:55.054590 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.054632 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:55.054794 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:55.055853 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:55.055860 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:55.055888 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:55.056031 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.056037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.057237 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.057243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.058264 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.058273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.059087 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.059090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.059194 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.059234 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:28:55.059358 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 14:28:55.059483 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":91,"threads_started":1}
I20260620 14:28:55.060464 16450 tablet_service.cc:1514] Processing DeleteTablet for tablet e8e85c2ba1ac420dae3d36945fab7da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34094
I20260620 14:28:55.061249 16525 tablet_replica.cc:330] stopping tablet replica
I20260620 14:28:55.061524 16525 raft_consensus.cc:2229] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:55.061745 16525 raft_consensus.cc:2258] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:55.062559 16525 ts_tablet_manager.cc:1905] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:55.065801 16525 ts_tablet_manager.cc:1918] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:55.065876 16525 log.cc:1196] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal/wals/e8e85c2ba1ac420dae3d36945fab7da3
I20260620 14:28:55.066143 16525 ts_tablet_manager.cc:1939] T e8e85c2ba1ac420dae3d36945fab7da3 P c5e3e035c975489da1933d543915e20b: Deleting consensus metadata
I20260620 14:28:55.067655 16313 catalog_manager.cc:4841] TS c5e3e035c975489da1933d543915e20b (127.0.98.1:45659): tablet e8e85c2ba1ac420dae3d36945fab7da3 (table test-table [id=41922daf8b3c40248702b41ddf905086]) successfully deleted
W20260620 14:28:55.147044 16520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:55.147298 16520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:55.147379 16520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:55.151278 16520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:55.151356 16520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:55.151438 16520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:28:55.156476 16520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33495
--tserver_master_addrs=127.0.98.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:55.157531 16520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:55.158483 16520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:55.167613 16531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:55.167718 16520 server_base.cc:1034] running on GCE node
W20260620 14:28:55.167658 16533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:55.167625 16530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:55.168226 16520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:55.168737 16520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:55.169925 16520 hybrid_clock.cc:648] HybridClock initialized: now 1781965735169913 us; error 24 us; skew 500 ppm
Jun 20 14:28:55 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:55.172501 16520 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:28:55.173420 16520 webserver.cc:460] Webserver started at http://127.0.98.2:33495/ using document root <none> and password file <none>
I20260620 14:28:55.173961 16520 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:55.174060 16520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:55.177656 16520 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:28:55.179481 16540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:55.180720 16520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:28:55.180840 16520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "1172c6ed388143669b766dcb6caa8e97"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "2374ec441f0763a15e069fe5a5eb979e"
server_key_iv: "fd35bf7a515aa3988ba9c5fc1c9d66e1"
server_key_version: "encryptionkey@0"
I20260620 14:28:55.181448 16520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:55.191883 16520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:55.194855 16520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:55.195040 16520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:55.195569 16520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:55.196300 16520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:55.196373 16520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:55.196440 16520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:55.196475 16520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:55.204954 16520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36451
I20260620 14:28:55.204988 16653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36451 every 8 connection(s)
I20260620 14:28:55.205935 16520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:28:55.211238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 16520
I20260620 14:28:55.211633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38301
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34885
--webserver_port=38231
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:28:55 dist-test-slave-pk3j krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:55.219331 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.207594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60197 (local address 127.0.98.62:34885)
0620 14:28:55.207732 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.207742 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.208461 (+ 719us) server_negotiation.cc:407] Connection header received
0620 14:28:55.209725 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.209728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.209784 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.209865 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:55.210958 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.211558 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.212737 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.212968 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.213082 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.215485 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:55.215491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:55.215498 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:55.215533 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:55.217224 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:55.217675 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:55.217681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:55.217685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:55.217780 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:55.218088 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:55.218093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:55.218096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:55.218349 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:55.218497 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.218949 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.219097 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:28:55.219861 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.207925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:60197)
0620 14:28:55.208314 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.208352 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:55.209519 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:55.210030 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:55.210036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:55.210361 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:55.210812 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.210825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.211679 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.211682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.212611 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.212620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.213115 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.213119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.213337 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.213903 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:55.213924 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:55.215306 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:55.217331 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:55.217335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:55.217345 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:55.217574 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:55.217878 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:55.217881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:55.217882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:55.217998 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:55.218486 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:55.218490 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:55.218696 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 14:28:55.218852 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 14:28:55.219084 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
I20260620 14:28:55.221019 16654 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34885
I20260620 14:28:55.221312 16654 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:55.221757 16654 heartbeater.cc:511] Master 127.0.98.62:34885 requested a full tablet report, sending...
I20260620 14:28:55.222774 16328 ts_manager.cc:194] Registered new tserver with Master: 1172c6ed388143669b766dcb6caa8e97 (127.0.98.2:36451)
I20260620 14:28:55.223313 16328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60197
W20260620 14:28:55.319712 16657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:28:55.320050 16657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:28:55.320120 16657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:28:55.323976 16657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:28:55.324029 16657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:28:55.324083 16657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:28:55.328960 16657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38231
--tserver_master_addrs=127.0.98.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:28:55.329958 16657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:28:55.330899 16657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:28:55.340030 16663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:55.340081 16662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:28:55.340072 16665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:28:55.340372 16657 server_base.cc:1034] running on GCE node
I20260620 14:28:55.340704 16657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:28:55.341289 16657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:28:55.342454 16657 hybrid_clock.cc:648] HybridClock initialized: now 1781965735342435 us; error 30 us; skew 500 ppm
Jun 20 14:28:55 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:28:55.344965 16657 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:28:55.345817 16657 webserver.cc:460] Webserver started at http://127.0.98.3:38231/ using document root <none> and password file <none>
I20260620 14:28:55.346349 16657 fs_manager.cc:359] Metadata directory not provided
I20260620 14:28:55.346453 16657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:28:55.350005 16657 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 14:28:55.351923 16672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:55.353020 16657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:28:55.353166 16657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "38d73562a73a4e3db87b6586ad4bb868"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "f3ad35002fe062b04b755490589c6b18"
server_key_iv: "0af4047ffec98c5751971a04adfc22ae"
server_key_version: "encryptionkey@0"
I20260620 14:28:55.353720 16657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:28:55.370290 16657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:28:55.372927 16657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:28:55.373106 16657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:28:55.373736 16657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:28:55.374507 16657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:28:55.374574 16657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:55.374636 16657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:28:55.374684 16657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:28:55.383428 16657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38301
I20260620 14:28:55.383463 16785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38301 every 8 connection(s)
I20260620 14:28:55.384443 16657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:28:55.387954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 16657
Jun 20 14:28:55 dist-test-slave-pk3j krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:28:55.397451 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.388567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.0.98.62:34885)
0620 14:28:55.388792 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.388796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.389003 (+ 207us) server_negotiation.cc:407] Connection header received
0620 14:28:55.389137 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.389145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.389179 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.389295 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:55.390170 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.390833 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.391743 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.392053 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.392246 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.394023 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:55.394027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:55.394030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:55.394053 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:55.395745 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:55.396228 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:55.396231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:55.396232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:55.396303 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:55.396589 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:55.396592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:55.396593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:55.396744 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:55.396884 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.397094 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.397239 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":77,"threads_started":1}
I20260620 14:28:55.399036 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.386127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54231 (local address 127.0.98.62:34885)
0620 14:28:55.386271 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.386276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.387069 (+ 793us) server_negotiation.cc:407] Connection header received
0620 14:28:55.388327 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.388333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.388395 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.388489 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:55.389671 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.390416 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.391650 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.391956 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.392028 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.394954 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:55.394958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:55.394960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:55.395000 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:55.396508 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:55.397473 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:55.397476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:55.397477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:55.397573 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:55.397973 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:55.397978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:55.397981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:55.398157 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:55.398278 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.398793 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.398897 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:28:55.399791 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.386528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:54231)
0620 14:28:55.386935 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.386972 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:55.388133 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:55.388671 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:55.388679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:55.389024 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:28:55.389527 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.389541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.390532 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.390536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.391543 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.391552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.392099 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.392105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.392367 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.392951 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:28:55.392967 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:28:55.394796 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:28:55.396927 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:55.396931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:55.396942 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:55.397375 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:55.397652 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:28:55.397655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:28:55.397657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:28:55.397858 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:28:55.398282 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:28:55.398287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:28:55.398561 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 14:28:55.398700 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 14:28:55.398918 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":109,"threads_started":1}
I20260620 14:28:55.400791 16786 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34885
I20260620 14:28:55.401018 16786 heartbeater.cc:464] Registering TS with master...
I20260620 14:28:55.401520 16786 heartbeater.cc:511] Master 127.0.98.62:34885 requested a full tablet report, sending...
I20260620 14:28:55.402402 16328 ts_manager.cc:194] Registered new tserver with Master: 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301)
I20260620 14:28:55.402917 16328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54231
I20260620 14:28:55.410735 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:28:55.414259 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.411307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.0.98.62:34885)
0620 14:28:55.411418 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.411422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.411590 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:28:55.411709 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.411711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.411794 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.411868 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:55.412050 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.412525 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.413326 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.413523 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.413595 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.413877 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:55.413990 (+ 113us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 14:28:55.414160 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59292: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 14:28:55.414337 16518 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59292: token expired
W20260620 14:28:55.414696 16255 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:34885)
I20260620 14:28:55.420955 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.415044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.0.98.62:34885)
0620 14:28:55.415142 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.415145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.415164 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:28:55.415212 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.415214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.415245 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.415306 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:28:55.415831 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.416280 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.416955 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.417136 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.417254 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.418501 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:28:55.418503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:28:55.418505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:28:55.418521 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:28:55.419681 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:55.420035 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:55.420038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:55.420039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:55.420107 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:28:55.420324 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:28:55.420326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:28:55.420327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:28:55.420499 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:28:55.420597 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.420771 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.420852 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:28:55.423059 16328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:28:55.426111 16328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:28:55.433288 16450 tablet_service.cc:1467] Processing CreateTablet for tablet 32160e6fd4e544d5a4f72d998656f2d3 (DEFAULT_TABLE table=test-table [id=abaaa8739e82431e978a5b543ef03096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:55.433902 16450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32160e6fd4e544d5a4f72d998656f2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:55.436530 16408 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Bootstrap starting.
I20260620 14:28:55.437078 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.432252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36451 (local address 127.0.0.1:52916)
0620 14:28:55.432431 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.432444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:55.432518 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:55.433383 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:55.433386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:55.433402 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:55.433500 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.433510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.434725 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.434730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.435673 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.435681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.436716 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.436735 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.436817 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.436839 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:28:55.436885 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:28:55.436958 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 14:28:55.437373 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.432460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38301 (local address 127.0.0.1:47694)
0620 14:28:55.432680 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.432691 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:55.432761 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:55.433738 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:55.433740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:55.433753 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:55.433838 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.433843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.434790 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.434793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.435833 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.435844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.436968 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.436971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.437047 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.437065 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:28:55.437142 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:28:55.437233 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260620 14:28:55.437435 16408 tablet_bootstrap.cc:654] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:55.437636 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.432797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.0.98.3:38301)
0620 14:28:55.433105 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.433114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.433132 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:28:55.433188 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.433192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.433432 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.433582 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:55.433965 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.434690 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.435951 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.436681 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.436772 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.436915 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.437382 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.437499 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":142,"threads_started":1}
I20260620 14:28:55.438272 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.432439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.0.98.2:36451)
0620 14:28:55.432882 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.432895 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.432929 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:28:55.433005 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.433010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.433241 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.433406 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:55.433597 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.434606 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.435794 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.436577 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.437345 (+ 768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.438012 (+ 667us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.438063 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.438145 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":126,"threads_started":1}
I20260620 14:28:55.439277 16408 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: No bootstrap required, opened a new log
I20260620 14:28:55.439366 16408 ts_tablet_manager.cc:1397] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:28:55.440069 16408 raft_consensus.cc:348] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.440178 16408 raft_consensus.cc:374] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:55.440217 16408 raft_consensus.cc:729] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e3e035c975489da1933d543915e20b, State: Initialized, Role: FOLLOWER
I20260620 14:28:55.440351 16408 consensus_queue.cc:260] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.440586 16408 ts_tablet_manager.cc:1428] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:28:55.440497 16720 tablet_service.cc:1467] Processing CreateTablet for tablet 32160e6fd4e544d5a4f72d998656f2d3 (DEFAULT_TABLE table=test-table [id=abaaa8739e82431e978a5b543ef03096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:55.441025 16588 tablet_service.cc:1467] Processing CreateTablet for tablet 32160e6fd4e544d5a4f72d998656f2d3 (DEFAULT_TABLE table=test-table [id=abaaa8739e82431e978a5b543ef03096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:28:55.441687 16720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32160e6fd4e544d5a4f72d998656f2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:55.442076 16588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32160e6fd4e544d5a4f72d998656f2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:28:55.448334 16799 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Bootstrap starting.
I20260620 14:28:55.448349 16800 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Bootstrap starting.
I20260620 14:28:55.450547 16800 tablet_bootstrap.cc:654] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:55.450925 16799 tablet_bootstrap.cc:654] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Neither blocks nor log segments found. Creating new log.
I20260620 14:28:55.451593 16800 log.cc:824] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:55.451592 16799 log.cc:824] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Log is configured to *not* fsync() on all Append() calls
I20260620 14:28:55.453333 16799 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: No bootstrap required, opened a new log
I20260620 14:28:55.453341 16800 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: No bootstrap required, opened a new log
I20260620 14:28:55.453533 16799 ts_tablet_manager.cc:1397] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:28:55.453542 16800 ts_tablet_manager.cc:1397] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:28:55.456395 16800 raft_consensus.cc:348] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.456394 16799 raft_consensus.cc:348] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.456596 16800 raft_consensus.cc:374] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:55.456663 16799 raft_consensus.cc:374] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:28:55.456714 16799 raft_consensus.cc:729] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1172c6ed388143669b766dcb6caa8e97, State: Initialized, Role: FOLLOWER
I20260620 14:28:55.456732 16800 raft_consensus.cc:729] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d73562a73a4e3db87b6586ad4bb868, State: Initialized, Role: FOLLOWER
I20260620 14:28:55.457156 16800 consensus_queue.cc:260] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.457156 16799 consensus_queue.cc:260] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.457863 16654 heartbeater.cc:503] Master 127.0.98.62:34885 was elected leader, sending a full tablet report...
I20260620 14:28:55.458065 16786 heartbeater.cc:503] Master 127.0.98.62:34885 was elected leader, sending a full tablet report...
I20260620 14:28:55.458184 16799 ts_tablet_manager.cc:1428] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:28:55.458180 16800 ts_tablet_manager.cc:1428] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260620 14:28:55.528200 16517 tablet.cc:2369] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:28:55.535671 16803 raft_consensus.cc:482] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:28:55.535868 16803 raft_consensus.cc:504] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.536978 16803 leader_election.cc:288] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301), c5e3e035c975489da1933d543915e20b (127.0.98.1:45659)
I20260620 14:28:55.540448 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.537284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39081 (local address 127.0.98.3:38301)
0620 14:28:55.537409 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.537413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.537427 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:28:55.537552 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.537556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.537607 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.537707 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:55.538022 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.538487 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.539497 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.539892 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.540003 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.540080 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.540271 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.540334 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:28:55.540448 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.537137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38301 (local address 127.0.98.2:39081)
0620 14:28:55.537324 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.537340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:55.537426 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:55.537755 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:55.537759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:55.537773 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:55.537900 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.537906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.538593 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.538599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.539375 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.539383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.540002 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.540006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.540133 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.540150 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:28:55.540252 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:28:55.540322 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":5}
I20260620 14:28:55.540592 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.537382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37599 (local address 127.0.98.1:45659)
0620 14:28:55.537522 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.537526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.537660 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:28:55.537768 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.537771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.537826 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.537911 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:28:55.538177 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.538643 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.539658 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.540063 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.540180 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.540238 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.540425 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.540493 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:28:55.540603 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.537251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45659 (local address 127.0.98.2:37599)
0620 14:28:55.537577 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.537586 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:28:55.537656 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:28:55.537931 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:28:55.537934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:28:55.537952 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:28:55.538066 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.538071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.538752 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.538758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.539539 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:28:55.539547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.540181 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.540185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:28:55.540288 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.540315 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:28:55.540400 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:28:55.540463 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":28,"thread_start_us":108,"threads_started":1}
I20260620 14:28:55.541152 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32160e6fd4e544d5a4f72d998656f2d3" candidate_uuid: "1172c6ed388143669b766dcb6caa8e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d73562a73a4e3db87b6586ad4bb868" is_pre_election: true
I20260620 14:28:55.541155 16470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32160e6fd4e544d5a4f72d998656f2d3" candidate_uuid: "1172c6ed388143669b766dcb6caa8e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e3e035c975489da1933d543915e20b" is_pre_election: true
I20260620 14:28:55.541460 16470 raft_consensus.cc:2454] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1172c6ed388143669b766dcb6caa8e97 in term 0.
I20260620 14:28:55.541468 16740 raft_consensus.cc:2454] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1172c6ed388143669b766dcb6caa8e97 in term 0.
I20260620 14:28:55.541965 16542 leader_election.cc:302] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1172c6ed388143669b766dcb6caa8e97, 38d73562a73a4e3db87b6586ad4bb868, c5e3e035c975489da1933d543915e20b; no voters:
I20260620 14:28:55.542196 16803 raft_consensus.cc:2781] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:28:55.542284 16803 raft_consensus.cc:482] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:28:55.542342 16803 raft_consensus.cc:3037] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:55.543324 16803 raft_consensus.cc:504] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.543646 16803 leader_election.cc:288] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 1 election: Requested vote from peers 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301), c5e3e035c975489da1933d543915e20b (127.0.98.1:45659)
I20260620 14:28:55.543958 16740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32160e6fd4e544d5a4f72d998656f2d3" candidate_uuid: "1172c6ed388143669b766dcb6caa8e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38d73562a73a4e3db87b6586ad4bb868"
I20260620 14:28:55.543989 16470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32160e6fd4e544d5a4f72d998656f2d3" candidate_uuid: "1172c6ed388143669b766dcb6caa8e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e3e035c975489da1933d543915e20b"
I20260620 14:28:55.544049 16740 raft_consensus.cc:3037] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:55.544129 16470 raft_consensus.cc:3037] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:28:55.545074 16740 raft_consensus.cc:2454] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1172c6ed388143669b766dcb6caa8e97 in term 1.
I20260620 14:28:55.545110 16470 raft_consensus.cc:2454] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1172c6ed388143669b766dcb6caa8e97 in term 1.
I20260620 14:28:55.545405 16543 leader_election.cc:302] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1172c6ed388143669b766dcb6caa8e97, 38d73562a73a4e3db87b6586ad4bb868; no voters:
I20260620 14:28:55.545571 16803 raft_consensus.cc:2781] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:28:55.545761 16803 raft_consensus.cc:686] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 LEADER]: Becoming Leader. State: Replica: 1172c6ed388143669b766dcb6caa8e97, State: Running, Role: LEADER
I20260620 14:28:55.546084 16803 consensus_queue.cc:237] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:28:55.548911 16327 catalog_manager.cc:5495] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1172c6ed388143669b766dcb6caa8e97 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1172c6ed388143669b766dcb6caa8e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } health_report { overall_health: UNKNOWN } } }
I20260620 14:28:55.563520 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.560547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.98.2:36451)
0620 14:28:55.560660 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:28:55.560664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:28:55.560675 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:28:55.560803 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:28:55.560805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:28:55.560854 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:28:55.560945 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:28:55.561125 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.561668 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.562394 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:28:55.562581 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:28:55.562650 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:28:55.562867 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:28:55.563233 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:28:55.563318 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:28:55.563363 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 14:28:55.563405 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:28:55.568930 16740 raft_consensus.cc:1261] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 1 FOLLOWER]: Refusing update from remote peer 1172c6ed388143669b766dcb6caa8e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:55.568979 16470 raft_consensus.cc:1261] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Refusing update from remote peer 1172c6ed388143669b766dcb6caa8e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:28:55.569655 16806 consensus_queue.cc:1035] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:55.569880 16803 consensus_queue.cc:1035] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:28:55.579007 16811 mvcc.cc:205] Tried to move back new op lower bound from 7298931652883349504 to 7298931652799311872. Current Snapshot: MvccSnapshot[applied={T|T < 7298931652883349504 or (T in {7298931652883349504})}]
I20260620 14:28:55.584511 16328 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:55.584695 16328 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:28:55.586879 16328 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Sending DeleteTablet for 3 replicas of tablet 32160e6fd4e544d5a4f72d998656f2d3
I20260620 14:28:55.587451 16588 tablet_service.cc:1514] Processing DeleteTablet for tablet 32160e6fd4e544d5a4f72d998656f2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52916
I20260620 14:28:55.587534 16720 tablet_service.cc:1514] Processing DeleteTablet for tablet 32160e6fd4e544d5a4f72d998656f2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47694
I20260620 14:28:55.587628 16450 tablet_service.cc:1514] Processing DeleteTablet for tablet 32160e6fd4e544d5a4f72d998656f2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:28:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34094
I20260620 14:28:55.587697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 16366
I20260620 14:28:55.588016 16818 tablet_replica.cc:330] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: stopping tablet replica
I20260620 14:28:55.588361 16817 tablet_replica.cc:330] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: stopping tablet replica
I20260620 14:28:55.588517 16818 raft_consensus.cc:2229] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:28:55.588672 16817 raft_consensus.cc:2229] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:28:55.588904 16818 raft_consensus.cc:2258] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:55.589048 16817 raft_consensus.cc:2258] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:28:55.590564 16818 ts_tablet_manager.cc:1905] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:28:55.593138 16818 ts_tablet_manager.cc:1918] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:28:55.593267 16818 log.cc:1196] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/32160e6fd4e544d5a4f72d998656f2d3
I20260620 14:28:55.593611 16818 ts_tablet_manager.cc:1939] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Deleting consensus metadata
I20260620 14:28:55.594889 16314 catalog_manager.cc:4841] TS 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301): tablet 32160e6fd4e544d5a4f72d998656f2d3 (table test-table [id=abaaa8739e82431e978a5b543ef03096]) successfully deleted
W20260620 14:28:55.595556 16313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45659 (error 108)
I20260620 14:28:55.596125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 16520
I20260620 14:28:55.597858 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.597541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45659 (local address 127.0.0.1:34098)
0620 14:28:55.597714 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.597789 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 14:28:55.598011 16313 catalog_manager.cc:4584] TS c5e3e035c975489da1933d543915e20b (127.0.98.1:45659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32160e6fd4e544d5a4f72d998656f2d3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45659: connect: Connection refused (error 111)
I20260620 14:28:55.602051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 16657
I20260620 14:28:55.603000 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:28:55.602196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36451 (local address 127.0.0.1:52938)
0620 14:28:55.602847 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 14:28:55.602910 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555}
I20260620 14:28:55.607327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 16295
I20260620 14:29:15.613911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34885
--webserver_interface=127.0.98.62
--webserver_port=40545
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:15.719641 16820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:15.719858 16820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:15.719921 16820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:15.723769 16820 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:29:15.723814 16820 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:29:15.723829 16820 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:29:15.723841 16820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:15.723858 16820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:15.723875 16820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:15.723891 16820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:15.723924 16820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:15.728655 16820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:15.729585 16820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:15.730468 16820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:15.738644 16826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:15.738692 16820 server_base.cc:1034] running on GCE node
W20260620 14:29:15.738695 16828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:15.738940 16825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:15.739459 16820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:15.740497 16820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:15.741669 16820 hybrid_clock.cc:648] HybridClock initialized: now 1781965755741642 us; error 42 us; skew 500 ppm
Jun 20 14:29:15 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:15.744324 16820 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:29:15.745123 16820 webserver.cc:460] Webserver started at http://127.0.98.62:40545/ using document root <none> and password file <none>
I20260620 14:29:15.745657 16820 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:15.745743 16820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:15.749387 16820 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 14:29:15.751401 16835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:15.752552 16820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:15.752686 16820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "032a22079ff945ddbe3a1d914408e56a"
format_stamp: "Formatted at 2026-06-20 14:28:33 on dist-test-slave-pk3j"
server_key: "3fbc15f9b3fd5ba31abd54b99b04b307"
server_key_iv: "93096cdadc4223b7414e5604158ad841"
server_key_version: "encryptionkey@0"
I20260620 14:29:15.753201 16820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:15.794132 16820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:15.797477 16820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:15.797695 16820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:15.804651 16820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34885
I20260620 14:29:15.804926 16887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34885 every 8 connection(s)
I20260620 14:29:15.805846 16820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:15.809990 16888 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:29:15.810639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 16820
I20260620 14:29:15.811065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45659
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34885
--webserver_port=44171
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:29:15.811712 16888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Bootstrap starting.
I20260620 14:29:15.835857 16888 log.cc:824] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:15.849632 16888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:29:15.850059 16888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Bootstrap complete.
I20260620 14:29:15.855545 16888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:29:15.856086 16888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032a22079ff945ddbe3a1d914408e56a, State: Initialized, Role: FOLLOWER
I20260620 14:29:15.856621 16888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:29:15.856763 16888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:15.856837 16888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:15.856946 16888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:29:15.859360 16888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:29:15.859699 16888 leader_election.cc:302] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032a22079ff945ddbe3a1d914408e56a; no voters:
I20260620 14:29:15.860018 16888 leader_election.cc:288] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:29:15.860150 16892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:29:15.860400 16892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [term 3 LEADER]: Becoming Leader. State: Replica: 032a22079ff945ddbe3a1d914408e56a, State: Running, Role: LEADER
I20260620 14:29:15.860684 16892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } }
I20260620 14:29:15.861232 16888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:15.862182 16894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032a22079ff945ddbe3a1d914408e56a. Latest consensus state: current_term: 3 leader_uuid: "032a22079ff945ddbe3a1d914408e56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } } }
I20260620 14:29:15.862293 16894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:15.862154 16893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "032a22079ff945ddbe3a1d914408e56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032a22079ff945ddbe3a1d914408e56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34885 } } }
I20260620 14:29:15.862386 16893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:15.862664 16901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:15.867914 16901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:15.868611 16901 catalog_manager.cc:1259] Loaded cluster ID: 2d939793587e48f98b92e559b99abfed
I20260620 14:29:15.868671 16901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:15.871243 16901 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:15.871975 16901 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Loaded TSK: 0
I20260620 14:29:15.872056 16901 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 032a22079ff945ddbe3a1d914408e56a: Loaded TSK: 1
F20260620 14:29:15.872812 16901 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb8f32c99ad google::LogMessage::Fail() at ??:0
@ 0x7fb8f32cc88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb8f32c94de google::LogMessage::Flush() at ??:0
@ 0x7fb8f32caa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb8f5f8d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb8f5f85f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb8fdc2e7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb8fdc0899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb8fdc09178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fb8fdc4adf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb8fdc6982b std::function<>::operator()() at ??:0
@ 0x7fb8fdc08b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb8fdc09f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb8fdc06579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb8fdc4a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb8f551b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb8f551bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb8f551d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb8f550c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb8f6b506db start_thread at ??:0
@ 0x7fb8f1dc761f clone at ??:0
W20260620 14:29:15.958519 16890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:15.958761 16890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:15.958803 16890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:15.963135 16890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:15.963177 16890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:15.963232 16890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:29:15.968807 16890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44171
--tserver_master_addrs=127.0.98.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:15.969771 16890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:15.970700 16890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:15.980470 16935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:15.980538 16890 server_base.cc:1034] running on GCE node
W20260620 14:29:15.980472 16936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:15.980736 16938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:15.981083 16890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:15.981743 16890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:15.982949 16890 hybrid_clock.cc:648] HybridClock initialized: now 1781965755982924 us; error 44 us; skew 500 ppm
Jun 20 14:29:15 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:15.985607 16890 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:29:15.986485 16890 webserver.cc:460] Webserver started at http://127.0.98.1:44171/ using document root <none> and password file <none>
I20260620 14:29:15.987022 16890 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:15.987082 16890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:15.990895 16890 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:29:15.993078 16945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:15.994345 16890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:29:15.994495 16890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "c5e3e035c975489da1933d543915e20b"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "71903ddc054320950128ef403ecf996c"
server_key_iv: "ce9ade81a818f14c792ab86119537783"
server_key_version: "encryptionkey@0"
I20260620 14:29:15.995035 16890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:16.027518 16890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:16.030829 16890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:16.031044 16890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:16.031651 16890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:16.032783 16952 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:29:16.038235 16890 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:29:16.038319 16890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 14:29:16.038419 16890 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:29:16.040036 16890 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:29:16.040105 16890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 14:29:16.040406 16952 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Bootstrap starting.
I20260620 14:29:16.050904 16890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45659
I20260620 14:29:16.050958 17059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45659 every 8 connection(s)
I20260620 14:29:16.051846 16890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:29:16.060179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 16890
I20260620 14:29:16.060632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36451
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34885
--webserver_port=33495
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:29:16.062083 16952 log.cc:824] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:16.070983 16952 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:29:16.071406 16952 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Bootstrap complete.
I20260620 14:29:16.072821 16952 ts_tablet_manager.cc:1397] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.007s
I20260620 14:29:16.076097 16952 raft_consensus.cc:348] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:29:16.076639 16952 raft_consensus.cc:729] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e3e035c975489da1933d543915e20b, State: Initialized, Role: FOLLOWER
I20260620 14:29:16.077145 16952 consensus_queue.cc:260] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:29:16.078189 16952 ts_tablet_manager.cc:1428] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260620 14:29:16.113488 17062 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:29:16.053741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:60869)
0620 14:29:16.054214 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.054267 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:16.055070 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:16.112553 (+ 57483us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":139,"threads_started":1}
W20260620 14:29:16.114071 17060 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34885 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34885: Client connection negotiation failed: client connection to 127.0.98.62:34885: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:29:16.114774 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:16.114513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:56233)
0620 14:29:16.114644 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.114713 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:29:16.115437 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:16.115233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:45921)
0620 14:29:16.115337 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.115383 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:29:16.115633 17060 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:29:16.169984 17063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:16.170212 17063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:16.170279 17063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:16.174252 17063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:16.174297 17063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:16.174384 17063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:29:16.179445 17063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33495
--tserver_master_addrs=127.0.98.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:16.180377 17063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:16.181427 17063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:29:16.190291 17063 server_base.cc:1034] running on GCE node
W20260620 14:29:16.190261 17070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:16.190256 17073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:16.190256 17071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:16.190961 17063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:16.191483 17063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:16.192636 17063 hybrid_clock.cc:648] HybridClock initialized: now 1781965756192603 us; error 42 us; skew 500 ppm
Jun 20 14:29:16 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:16.195123 17063 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:29:16.195883 17063 webserver.cc:460] Webserver started at http://127.0.98.2:33495/ using document root <none> and password file <none>
I20260620 14:29:16.196416 17063 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:16.196471 17063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:16.200171 17063 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:16.202124 17080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:16.203239 17063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:16.203356 17063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "1172c6ed388143669b766dcb6caa8e97"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "2374ec441f0763a15e069fe5a5eb979e"
server_key_iv: "fd35bf7a515aa3988ba9c5fc1c9d66e1"
server_key_version: "encryptionkey@0"
I20260620 14:29:16.203852 17063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:16.231241 17063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:16.233997 17063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:16.234185 17063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:16.234732 17063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:16.235689 17087 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:29:16.239519 17063 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:29:16.239580 17063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 14:29:16.239658 17063 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:29:16.241218 17063 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:29:16.241273 17063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:16.241537 17087 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Bootstrap starting.
I20260620 14:29:16.251803 17063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36451
I20260620 14:29:16.251992 17194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36451 every 8 connection(s)
I20260620 14:29:16.252950 17063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:29:16.256465 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:16.254917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:50417)
0620 14:29:16.255434 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.255583 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":104,"threads_started":1}
W20260620 14:29:16.256889 17195 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34885 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34885: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
I20260620 14:29:16.257498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 17063
I20260620 14:29:16.257507 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:16.257279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:47753)
0620 14:29:16.257395 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.257447 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:29:16.258005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38301
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34885
--webserver_port=38231
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:29:16.258239 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:16.257950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:57557)
0620 14:29:16.258091 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.258154 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:29:16.258407 17195 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:29:16.260488 17087 log.cc:824] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:16.266847 17087 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:29:16.267184 17087 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Bootstrap complete.
I20260620 14:29:16.268261 17087 ts_tablet_manager.cc:1397] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20260620 14:29:16.270777 17087 raft_consensus.cc:348] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:29:16.271159 17087 raft_consensus.cc:729] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1172c6ed388143669b766dcb6caa8e97, State: Initialized, Role: FOLLOWER
I20260620 14:29:16.271611 17087 consensus_queue.cc:260] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:29:16.272476 17087 ts_tablet_manager.cc:1428] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260620 14:29:16.272724 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:16.272479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:53517)
0620 14:29:16.272600 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.272657 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:29:16.273353 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:16.273126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:43707)
0620 14:29:16.273260 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.273302 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:29:16.363806 17198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:16.364061 17198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:16.364128 17198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:16.368094 17198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:16.368175 17198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:16.368261 17198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:29:16.373168 17198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38231
--tserver_master_addrs=127.0.98.62:34885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:16.374161 17198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:16.375180 17198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:16.384452 17206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:16.384457 17205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:16.384457 17208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:16.384600 17198 server_base.cc:1034] running on GCE node
I20260620 14:29:16.385131 17198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:16.385720 17198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:16.386904 17198 hybrid_clock.cc:648] HybridClock initialized: now 1781965756386885 us; error 32 us; skew 500 ppm
Jun 20 14:29:16 dist-test-slave-pk3j krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:16.389451 17198 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:29:16.390172 17198 webserver.cc:460] Webserver started at http://127.0.98.3:38231/ using document root <none> and password file <none>
I20260620 14:29:16.390642 17198 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:16.390708 17198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:16.394212 17198 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:29:16.396042 17215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:16.397200 17198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:16.397367 17198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "38d73562a73a4e3db87b6586ad4bb868"
format_stamp: "Formatted at 2026-06-20 14:28:34 on dist-test-slave-pk3j"
server_key: "f3ad35002fe062b04b755490589c6b18"
server_key_iv: "0af4047ffec98c5751971a04adfc22ae"
server_key_version: "encryptionkey@0"
I20260620 14:29:16.397889 17198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:16.427367 17198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:16.430217 17198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:16.430419 17198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:16.431001 17198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:16.431782 17198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:16.431830 17198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:16.431916 17198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:16.431950 17198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:16.440340 17198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38301
I20260620 14:29:16.440363 17328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38301 every 8 connection(s)
I20260620 14:29:16.441309 17198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:29:16.444759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 17198
I20260620 14:29:16.445062 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:16.443310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:38673)
0620 14:29:16.444113 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.444209 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":504,"threads_started":1}
W20260620 14:29:16.445713 17329 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34885 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34885: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
I20260620 14:29:16.446266 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:16.446082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:42455)
0620 14:29:16.446179 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.446220 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:29:16.446846 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:16.446670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:59285)
0620 14:29:16.446763 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:16.446802 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:29:16.447054 17329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:29:17.116636 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:17.116242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:50909)
0620 14:29:17.116479 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:17.116531 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260620 14:29:17.274466 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:17.274090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:49921)
0620 14:29:17.274325 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:17.274374 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260620 14:29:17.448084 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:17.447648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:58643)
0620 14:29:17.447937 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:17.447990 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":77,"threads_started":1}
I20260620 14:29:17.823707 17336 raft_consensus.cc:482] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:17.823916 17336 raft_consensus.cc:504] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:29:17.825021 17336 leader_election.cc:288] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301), c5e3e035c975489da1933d543915e20b (127.0.98.1:45659)
Jun 20 14:29:17 dist-test-slave-pk3j krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
Jun 20 14:29:17 dist-test-slave-pk3j krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 14:29:17.838480 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:17.825238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45659 (local address 127.0.98.2:58777)
0620 14:29:17.825911 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:17.825926 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:17.826077 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:17.826859 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:17.826868 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:17.827656 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:17.828192 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:17.828205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:17.829500 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:17.829504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:17.830365 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:17.830374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:17.831059 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:17.831063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:17.831304 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:17.832056 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:17.832069 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:17.833619 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:17.835980 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:17.835985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:17.835995 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:17.836325 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:17.836736 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:17.836739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:17.836741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:17.836942 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:17.837647 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:17.837653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:17.837930 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 14:29:17.838111 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 14:29:17.838342 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":523,"spinlock_wait_cycles":8448,"thread_start_us":75,"threads_started":1}
I20260620 14:29:17.838613 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:17.825388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58777 (local address 127.0.98.1:45659)
0620 14:29:17.825681 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:17.825691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:17.826055 (+ 364us) server_negotiation.cc:407] Connection header received
0620 14:29:17.826392 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:17.826396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:17.826584 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:17.826715 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:17.828358 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:17.829369 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:17.830486 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:17.830941 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:17.831092 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:17.833822 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:17.833828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:17.833839 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:17.833862 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:17.835870 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:17.836462 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:17.836470 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:17.836477 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:17.836633 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:17.837045 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:17.837049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:17.837051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:17.837540 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:17.837669 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:17.838223 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 14:29:17.838436 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260620 14:29:17.839100 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:17.825238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38301 (local address 127.0.98.2:47197)
0620 14:29:17.825576 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:17.825607 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:17.826076 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:17.827679 (+ 1603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:17.827681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:17.827839 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:17.828190 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:17.828204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:17.829511 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:17.829515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:17.830366 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:17.830375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:17.831103 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:17.831107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:17.831305 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:17.831976 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:17.832010 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:17.833951 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:17.836157 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:17.836160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:17.836163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:17.836400 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:17.836736 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:17.836739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:17.836741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:17.836942 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:17.838640 (+ 1698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:17.838643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:17.838728 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 14:29:17.838838 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:29:17.838958 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":111,"spinlock_wait_cycles":8576,"thread_start_us":70,"threads_started":1}
I20260620 14:29:17.839402 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:17.825405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47197 (local address 127.0.98.3:38301)
0620 14:29:17.825856 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:17.825880 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:17.825925 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:29:17.827035 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:17.827043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:17.827427 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:17.827722 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:17.828359 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:17.829366 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:17.830523 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:17.830962 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:17.831140 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:17.834131 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:17.834139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:17.834157 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:17.834205 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:17.836063 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:17.836500 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:17.836506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:17.836511 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:17.836633 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:17.837045 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:17.837048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:17.837050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:17.837540 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:17.837669 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:17.838970 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0620 14:29:17.839203 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":115,"threads_started":1}
I20260620 14:29:17.839847 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32160e6fd4e544d5a4f72d998656f2d3" candidate_uuid: "1172c6ed388143669b766dcb6caa8e97" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c5e3e035c975489da1933d543915e20b" is_pre_election: true
I20260620 14:29:17.840154 17014 raft_consensus.cc:2454] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1172c6ed388143669b766dcb6caa8e97 in term 1.
I20260620 14:29:17.841181 17082 leader_election.cc:302] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1172c6ed388143669b766dcb6caa8e97, c5e3e035c975489da1933d543915e20b; no voters:
W20260620 14:29:17.841368 17083 leader_election.cc:341] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301): Not found: Tablet not found: 32160e6fd4e544d5a4f72d998656f2d3
I20260620 14:29:17.841487 17336 raft_consensus.cc:2781] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:29:17.841578 17336 raft_consensus.cc:482] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:29:17.841634 17336 raft_consensus.cc:3037] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:29:17.843638 17336 raft_consensus.cc:504] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:29:17.843993 17336 leader_election.cc:288] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 2 election: Requested vote from peers 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301), c5e3e035c975489da1933d543915e20b (127.0.98.1:45659)
I20260620 14:29:17.844416 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32160e6fd4e544d5a4f72d998656f2d3" candidate_uuid: "1172c6ed388143669b766dcb6caa8e97" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c5e3e035c975489da1933d543915e20b"
I20260620 14:29:17.844563 17014 raft_consensus.cc:3037] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 1 FOLLOWER]: Advancing to term 2
W20260620 14:29:17.844727 17083 leader_election.cc:341] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301): Not found: Tablet not found: 32160e6fd4e544d5a4f72d998656f2d3
I20260620 14:29:17.846485 17014 raft_consensus.cc:2454] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1172c6ed388143669b766dcb6caa8e97 in term 2.
I20260620 14:29:17.846755 17082 leader_election.cc:302] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1172c6ed388143669b766dcb6caa8e97, c5e3e035c975489da1933d543915e20b; no voters: 38d73562a73a4e3db87b6586ad4bb868
I20260620 14:29:17.846904 17336 raft_consensus.cc:2781] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:29:17.847067 17336 raft_consensus.cc:686] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [term 2 LEADER]: Becoming Leader. State: Replica: 1172c6ed388143669b766dcb6caa8e97, State: Running, Role: LEADER
I20260620 14:29:17.847388 17336 consensus_queue.cc:237] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:29:17.847675 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:17.847462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:34489)
0620 14:29:17.847570 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:17.847614 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:29:18.117882 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:18.117490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:51223)
0620 14:29:18.117723 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:18.117773 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
W20260620 14:29:18.400838 17083 consensus_peers.cc:487] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 -> Peer 38d73562a73a4e3db87b6586ad4bb868 (127.0.98.3:38301): Couldn't send request to peer 38d73562a73a4e3db87b6586ad4bb868. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32160e6fd4e544d5a4f72d998656f2d3. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:29:18.449173 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:18.448762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:50507)
0620 14:29:18.449036 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:18.449084 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":115,"threads_started":1}
I20260620 14:29:18.450081 17014 raft_consensus.cc:1261] T 32160e6fd4e544d5a4f72d998656f2d3 P c5e3e035c975489da1933d543915e20b [term 2 FOLLOWER]: Refusing update from remote peer 1172c6ed388143669b766dcb6caa8e97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 14:29:18.450773 17344 consensus_queue.cc:1035] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 14:29:18.452679 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:18.452451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:53313)
0620 14:29:18.452567 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:18.452619 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:29:18.455323 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:18.454995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:59705)
0620 14:29:18.455171 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:18.455230 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":52,"threads_started":1}
I20260620 14:29:18.455529 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:18.455278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:39623)
0620 14:29:18.455404 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:18.455460 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:29:18.791430 17352 ts_tablet_manager.cc:927] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Initiating tablet copy from peer 1172c6ed388143669b766dcb6caa8e97 (127.0.98.2:36451)
I20260620 14:29:18.791980 17352 tablet_copy_client.cc:316] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:36451
Jun 20 14:29:18 dist-test-slave-pk3j krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 14:29:18.801729 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:18.792482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36451 (local address 127.0.98.3:47447)
0620 14:29:18.792616 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:18.792637 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:18.792779 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:18.793432 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:18.793437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:18.793783 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:18.793890 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:18.793906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:18.794801 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:18.794803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:18.795405 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:18.795412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:18.795940 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:18.795942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:18.796019 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:18.796624 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:18.796635 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:18.798203 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:18.799992 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:18.800001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:18.800007 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:18.800225 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:18.800525 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:18.800529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:18.800530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:18.800614 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:18.801085 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:18.801091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:18.801235 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:29:18.801399 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 14:29:18.801564 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:29:18.801784 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:18.792629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47447 (local address 127.0.98.2:36451)
0620 14:29:18.792935 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:18.792945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:18.792965 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:29:18.793011 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:18.793015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:18.793294 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:18.793413 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:18.794006 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:18.794709 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:18.795502 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:18.795839 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:18.795918 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:18.798350 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:18.798365 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:18.798375 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:18.798398 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:18.799881 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:18.800343 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:18.800352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:18.800354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:18.800426 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:18.800729 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:18.800737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:18.800739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:18.800963 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:18.801100 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:18.801483 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 14:29:18.801614 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":111,"threads_started":1}
I20260620 14:29:18.802758 17169 tablet_copy_service.cc:140] P 1172c6ed388143669b766dcb6caa8e97: Received BeginTabletCopySession request for tablet 32160e6fd4e544d5a4f72d998656f2d3 from peer 38d73562a73a4e3db87b6586ad4bb868 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47447)
I20260620 14:29:18.802932 17169 tablet_copy_service.cc:161] P 1172c6ed388143669b766dcb6caa8e97: Beginning new tablet copy session on tablet 32160e6fd4e544d5a4f72d998656f2d3 from peer 38d73562a73a4e3db87b6586ad4bb868 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47447: session id = 38d73562a73a4e3db87b6586ad4bb868-32160e6fd4e544d5a4f72d998656f2d3
I20260620 14:29:18.804081 17169 tablet_copy_source_session.cc:205] T 32160e6fd4e544d5a4f72d998656f2d3 P 1172c6ed388143669b766dcb6caa8e97: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:29:18.808058 17352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32160e6fd4e544d5a4f72d998656f2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:18.814142 17352 tablet_copy_client.cc:791] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: tablet copy: Starting download of 0 data blocks...
I20260620 14:29:18.814332 17352 tablet_copy_client.cc:655] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: tablet copy: Starting download of 1 WAL segments...
I20260620 14:29:18.815893 17352 tablet_copy_client.cc:531] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:29:18.817448 17352 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Bootstrap starting.
I20260620 14:29:18.822618 17352 log.cc:824] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:18.828416 17352 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:29:18.828696 17352 tablet_bootstrap.cc:492] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Bootstrap complete.
I20260620 14:29:18.828987 17352 ts_tablet_manager.cc:1397] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 14:29:18.831302 17352 raft_consensus.cc:348] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:29:18.831524 17352 raft_consensus.cc:729] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d73562a73a4e3db87b6586ad4bb868, State: Initialized, Role: FOLLOWER
I20260620 14:29:18.831967 17352 consensus_queue.cc:260] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172c6ed388143669b766dcb6caa8e97" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36451 } } peers { permanent_uuid: "38d73562a73a4e3db87b6586ad4bb868" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38301 } } peers { permanent_uuid: "c5e3e035c975489da1933d543915e20b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45659 } }
I20260620 14:29:18.832774 17352 ts_tablet_manager.cc:1428] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260620 14:29:18.833029 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:18.832805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:58923)
0620 14:29:18.832915 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:18.832969 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:29:18.833452 17169 tablet_copy_service.cc:342] P 1172c6ed388143669b766dcb6caa8e97: Request end of tablet copy session 38d73562a73a4e3db87b6586ad4bb868-32160e6fd4e544d5a4f72d998656f2d3 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47447
I20260620 14:29:18.833560 17169 tablet_copy_service.cc:433] P 1172c6ed388143669b766dcb6caa8e97: ending tablet copy session 38d73562a73a4e3db87b6586ad4bb868-32160e6fd4e544d5a4f72d998656f2d3 on tablet 32160e6fd4e544d5a4f72d998656f2d3 with peer 38d73562a73a4e3db87b6586ad4bb868
I20260620 14:29:18.833649 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:18.833433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:36791)
0620 14:29:18.833540 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:18.833588 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:29:19.375190 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:19.374856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:54503)
0620 14:29:19.375062 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:19.375104 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":52,"threads_started":1}
I20260620 14:29:19.375234 17283 raft_consensus.cc:1203] T 32160e6fd4e544d5a4f72d998656f2d3 P 38d73562a73a4e3db87b6586ad4bb868 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 14:29:19.376662 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:19.376360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:34857)
0620 14:29:19.376549 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:19.376593 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260620 14:29:19.456547 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:19.456186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:42209)
0620 14:29:19.456413 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:19.456461 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260620 14:29:20.376278 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:20.375914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:37755)
0620 14:29:20.376135 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:20.376187 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
I20260620 14:29:20.377637 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:20.377274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:56779)
0620 14:29:20.377514 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:20.377563 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":56,"threads_started":1}
I20260620 14:29:20.457775 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:20.457378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:58795)
0620 14:29:20.457629 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:20.457687 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260620 14:29:21.377490 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:21.377075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:45597)
0620 14:29:21.377343 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:21.377400 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
I20260620 14:29:21.378304 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:21.378009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:43119)
0620 14:29:21.378148 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:21.378217 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:29:21.378736 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:21.378290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:49269)
0620 14:29:21.378614 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:21.378660 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":65,"threads_started":1}
I20260620 14:29:21.459049 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:21.458636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:46369)
0620 14:29:21.458908 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:21.458961 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260620 14:29:22.379382 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:22.378931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:40619)
0620 14:29:22.379240 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:22.379292 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260620 14:29:22.380064 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:22.379668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:53221)
0620 14:29:22.379914 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:22.379973 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260620 14:29:22.460279 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:22.459871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:34467)
0620 14:29:22.460117 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:22.460165 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260620 14:29:23.380904 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:23.380472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:39893)
0620 14:29:23.380751 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:23.380802 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
I20260620 14:29:23.381129 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:23.380714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:41029)
0620 14:29:23.380937 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:23.381018 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260620 14:29:23.461479 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:23.461081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:52603)
0620 14:29:23.461327 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:23.461387 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":81,"threads_started":1}
I20260620 14:29:24.382040 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.381671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:58901)
0620 14:29:24.381902 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.381950 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
I20260620 14:29:24.382112 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.381755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:48773)
0620 14:29:24.381977 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.382030 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":59,"threads_started":1}
I20260620 14:29:24.462715 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:24.462202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:56443)
0620 14:29:24.462501 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:24.462569 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
I20260620 14:29:25.383175 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:25.382751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:42241)
0620 14:29:25.383010 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:25.383080 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
I20260620 14:29:25.383188 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:25.382821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:53887)
0620 14:29:25.383057 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:25.383105 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260620 14:29:25.463830 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:25.463433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:49623)
0620 14:29:25.463676 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:25.463731 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260620 14:29:26.384454 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:26.384037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:42355)
0620 14:29:26.384310 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:26.384362 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":70,"threads_started":1}
I20260620 14:29:26.384671 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:26.384030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:40185)
0620 14:29:26.384339 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:26.384390 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260620 14:29:26.465055 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:26.464679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:51395)
0620 14:29:26.464919 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:26.464969 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260620 14:29:27.385706 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:27.385176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:37101)
0620 14:29:27.385560 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:27.385613 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":118,"threads_started":1}
I20260620 14:29:27.386152 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:27.385660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:53049)
0620 14:29:27.386025 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:27.386075 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260620 14:29:27.466132 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:27.465748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:44045)
0620 14:29:27.465991 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:27.466042 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260620 14:29:28.386853 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:28.386450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:57781)
0620 14:29:28.386711 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:28.386759 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
I20260620 14:29:28.387159 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:28.386786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:57405)
0620 14:29:28.387025 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:28.387083 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":109,"threads_started":1}
I20260620 14:29:28.467180 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:28.466810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:57029)
0620 14:29:28.467045 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:28.467094 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
I20260620 14:29:29.388119 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:29.387730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:54211)
0620 14:29:29.387983 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:29.388030 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20260620 14:29:29.388177 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:29.387787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:39547)
0620 14:29:29.388024 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:29.388094 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":107,"threads_started":1}
I20260620 14:29:29.468220 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:29.467870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:57391)
0620 14:29:29.468083 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:29.468132 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260620 14:29:30.389297 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:30.388860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:55101)
0620 14:29:30.389106 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:30.389152 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260620 14:29:30.389349 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:30.388896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:49157)
0620 14:29:30.389150 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:30.389257 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":107,"threads_started":1}
I20260620 14:29:30.469336 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:30.468919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.1:44395)
0620 14:29:30.469173 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:30.469248 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260620 14:29:31.390599 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:31.390208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.2:48471)
0620 14:29:31.390456 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:31.390508 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":66,"threads_started":1}
I20260620 14:29:31.390635 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:31.390196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34885 (local address 127.0.98.3:58965)
0620 14:29:31.390476 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:31.390526 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
I20260620 14:29:31.455506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 16890
I20260620 14:29:31.461279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 17063
I20260620 14:29:31.466740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 17198
2026-06-20T14:29:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58984 ms)
[----------] 30 tests from SecurityITest (234488 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:31 dist-test-slave-pk3j krb5kdc[17404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:31 dist-test-slave-pk3j krb5kdc[17404](info): set up 2 sockets
Jun 20 14:29:31 dist-test-slave-pk3j krb5kdc[17404](info): commencing operation
krb5kdc: starting...
W20260620 14:29:32.488961 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:32 dist-test-slave-pk3j krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:29:32.599263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45625
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:32.703197 17417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:32.703490 17417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:32.703528 17417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:32.707212 17417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:32.707253 17417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:32.707271 17417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:32.707286 17417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:32.707300 17417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:32.712040 17417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:32.712869 17417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:32.713804 17417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:32.721688 17422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:32.721740 17417 server_base.cc:1034] running on GCE node
W20260620 14:29:32.721699 17423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:32.721705 17425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:32.722414 17417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:32.723367 17417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:32.724565 17417 hybrid_clock.cc:648] HybridClock initialized: now 1781965772724554 us; error 42 us; skew 500 ppm
Jun 20 14:29:32 dist-test-slave-pk3j krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:32.727059 17417 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:29:32.727839 17417 webserver.cc:460] Webserver started at http://127.0.98.62:36051/ using document root <none> and password file <none>
I20260620 14:29:32.728353 17417 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:32.728448 17417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:32.728662 17417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:32.730520 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "c4b013d7e01e4363b1d79a872cb178bb"
format_stamp: "Formatted at 2026-06-20 14:29:32 on dist-test-slave-pk3j"
server_key: "35a2d2495dbb5044a7c4dc8788669a03"
server_key_iv: "9007fe302aabb608a1b245e584a8fdc5"
server_key_version: "encryptionkey@0"
I20260620 14:29:32.731047 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4b013d7e01e4363b1d79a872cb178bb"
format_stamp: "Formatted at 2026-06-20 14:29:32 on dist-test-slave-pk3j"
server_key: "35a2d2495dbb5044a7c4dc8788669a03"
server_key_iv: "9007fe302aabb608a1b245e584a8fdc5"
server_key_version: "encryptionkey@0"
I20260620 14:29:32.734586 17417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:29:32.736725 17432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:32.737920 17417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:32.738065 17417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "c4b013d7e01e4363b1d79a872cb178bb"
format_stamp: "Formatted at 2026-06-20 14:29:32 on dist-test-slave-pk3j"
server_key: "35a2d2495dbb5044a7c4dc8788669a03"
server_key_iv: "9007fe302aabb608a1b245e584a8fdc5"
server_key_version: "encryptionkey@0"
I20260620 14:29:32.738183 17417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:32.749075 17417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:32.751582 17417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:32.751770 17417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:32.758067 17417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45625
I20260620 14:29:32.758126 17484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45625 every 8 connection(s)
I20260620 14:29:32.758944 17417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:32.761466 17485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:32.764252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 17417
I20260620 14:29:32.764376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:32.764688 392 external_mini_cluster.cc:1428] Setting key 1f88f86377917a6e8deef6ada24cb029
I20260620 14:29:32.766983 17485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb: Bootstrap starting.
I20260620 14:29:32.769300 17485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:32.770165 17485 log.cc:824] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:32.772171 17485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb: No bootstrap required, opened a new log
Jun 20 14:29:32 dist-test-slave-pk3j krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:32.776057 17485 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b013d7e01e4363b1d79a872cb178bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } }
I20260620 14:29:32.776311 17485 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:32.776374 17485 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4b013d7e01e4363b1d79a872cb178bb, State: Initialized, Role: FOLLOWER
I20260620 14:29:32.776841 17485 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b013d7e01e4363b1d79a872cb178bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } }
I20260620 14:29:32.776993 17485 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:32.777093 17485 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:32.777226 17485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:32.778223 17485 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b013d7e01e4363b1d79a872cb178bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } }
I20260620 14:29:32.778544 17485 leader_election.cc:302] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4b013d7e01e4363b1d79a872cb178bb; no voters:
I20260620 14:29:32.778843 17485 leader_election.cc:288] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:32.778965 17490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:32.779199 17490 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [term 1 LEADER]: Becoming Leader. State: Replica: c4b013d7e01e4363b1d79a872cb178bb, State: Running, Role: LEADER
I20260620 14:29:32.779289 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:32.765925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.0.98.62:45625)
0620 14:29:32.766298 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:32.766321 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:32.766353 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:29:32.766909 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:32.766924 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:32.767271 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:32.767530 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:32.768347 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:32.769307 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:32.770490 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:32.771111 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:32.771227 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:32.773869 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:32.773877 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:32.773894 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:32.773922 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:32.776354 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:32.776834 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:32.776840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:32.776844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:32.776976 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:32.777291 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:32.777295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:32.777297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:32.777736 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:32.777930 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:32.778135 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:29:32.778415 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":136,"threads_started":1}
I20260620 14:29:32.779536 17490 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b013d7e01e4363b1d79a872cb178bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } }
I20260620 14:29:32.779984 17485 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:32.782243 17492 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4b013d7e01e4363b1d79a872cb178bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b013d7e01e4363b1d79a872cb178bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } } }
I20260620 14:29:32.782377 17492 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:32.782708 17491 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4b013d7e01e4363b1d79a872cb178bb. Latest consensus state: current_term: 1 leader_uuid: "c4b013d7e01e4363b1d79a872cb178bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b013d7e01e4363b1d79a872cb178bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45625 } } }
I20260620 14:29:32.782797 17491 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb [sys.catalog]: This master's current role is: LEADER
W20260620 14:29:32.784795 17504 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:29:32.784869 17504 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:29:32.784924 17505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:32.785748 17505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:32.790637 17505 catalog_manager.cc:1347] Generated new cluster ID: ef8fb17850cb48b782d924a9031c38de
I20260620 14:29:32.790709 17505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:32.802904 17505 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:32.803717 17505 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:32.814461 17505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb: Generated new TSK 0
I20260620 14:29:32.815147 17505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:29:32.842150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45625
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:32.948712 17511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:32.948936 17511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:32.949007 17511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:32.952932 17511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:32.952982 17511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:32.953102 17511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:29:32.958217 17511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:32.959147 17511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:32.960094 17511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:32.968899 17516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:32.968935 17519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:32.968928 17517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:32.969082 17511 server_base.cc:1034] running on GCE node
I20260620 14:29:32.969568 17511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:32.970157 17511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:32.971338 17511 hybrid_clock.cc:648] HybridClock initialized: now 1781965772971328 us; error 28 us; skew 500 ppm
Jun 20 14:29:32 dist-test-slave-pk3j krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:32.974025 17511 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:29:32.974812 17511 webserver.cc:460] Webserver started at http://127.0.98.1:42527/ using document root <none> and password file <none>
I20260620 14:29:32.975296 17511 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:32.975350 17511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:32.975518 17511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:32.977442 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "8154ab6054f64745aee342fc12e91b51"
format_stamp: "Formatted at 2026-06-20 14:29:32 on dist-test-slave-pk3j"
server_key: "5618caa7534cb922cd6752537a697dea"
server_key_iv: "f70f1add450e1144074a4294d54f3d09"
server_key_version: "encryptionkey@0"
I20260620 14:29:32.977905 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8154ab6054f64745aee342fc12e91b51"
format_stamp: "Formatted at 2026-06-20 14:29:32 on dist-test-slave-pk3j"
server_key: "5618caa7534cb922cd6752537a697dea"
server_key_iv: "f70f1add450e1144074a4294d54f3d09"
server_key_version: "encryptionkey@0"
I20260620 14:29:32.981321 17511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:29:32.983490 17526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:32.984678 17511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:32.984789 17511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "8154ab6054f64745aee342fc12e91b51"
format_stamp: "Formatted at 2026-06-20 14:29:32 on dist-test-slave-pk3j"
server_key: "5618caa7534cb922cd6752537a697dea"
server_key_iv: "f70f1add450e1144074a4294d54f3d09"
server_key_version: "encryptionkey@0"
I20260620 14:29:32.984887 17511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:33.009956 17511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:33.013171 17511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:33.013430 17511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:33.013989 17511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:33.014810 17511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:33.014863 17511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:33.014928 17511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:33.014961 17511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:33.023846 17511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45067
I20260620 14:29:33.023885 17639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45067 every 8 connection(s)
I20260620 14:29:33.024677 17511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:29:33.027664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 17511
I20260620 14:29:33.027786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:29:33.028077 392 external_mini_cluster.cc:1428] Setting key 7c32e08d79669308e74d7879504357c0
Jun 20 14:29:33 dist-test-slave-pk3j krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:29:33.038264 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.026396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55155 (local address 127.0.98.62:45625)
0620 14:29:33.026518 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.026522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.027278 (+ 756us) server_negotiation.cc:407] Connection header received
0620 14:29:33.028210 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.028213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.028277 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.028358 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:33.029657 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.030193 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.031362 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.031618 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.031730 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.034541 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:33.034544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:33.034546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:33.034569 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:33.036050 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:33.036664 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:33.036667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:33.036668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:33.036739 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:33.037257 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:33.037261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:33.037262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:33.037437 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:33.037514 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.038037 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.038136 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:29:33.039148 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.026731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45625 (local address 127.0.98.1:55155)
0620 14:29:33.027146 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:33.027185 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:33.028016 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:33.028550 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:33.028559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:33.028944 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:33.029519 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.029532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.030321 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.030326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.031235 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.031244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.031771 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.031775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.031945 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.032512 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:33.032527 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:33.034331 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:33.036169 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:33.036178 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:33.036194 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:33.036560 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:33.036823 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:33.036826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:33.036828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:33.037002 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:33.037538 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:33.037542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:33.037752 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 14:29:33.037976 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 14:29:33.038208 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260620 14:29:33.040385 17640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45625
I20260620 14:29:33.040648 17640 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:33.041347 17640 heartbeater.cc:511] Master 127.0.98.62:45625 requested a full tablet report, sending...
I20260620 14:29:33.042985 17449 ts_manager.cc:194] Registered new tserver with Master: 8154ab6054f64745aee342fc12e91b51 (127.0.98.1:45067)
I20260620 14:29:33.044094 17449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:29:33.053125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45625
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:33.157027 17645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:33.157306 17645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:33.157382 17645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:33.161247 17645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:33.161329 17645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:33.161418 17645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:29:33.166385 17645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:33.167308 17645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:33.168246 17645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:33.177301 17650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:33.177302 17651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:33.177400 17645 server_base.cc:1034] running on GCE node
W20260620 14:29:33.177302 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:33.178011 17645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:33.178586 17645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:33.179790 17645 hybrid_clock.cc:648] HybridClock initialized: now 1781965773179764 us; error 47 us; skew 500 ppm
Jun 20 14:29:33 dist-test-slave-pk3j krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:33.182580 17645 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:29:33.183349 17645 webserver.cc:460] Webserver started at http://127.0.98.2:40591/ using document root <none> and password file <none>
I20260620 14:29:33.183871 17645 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:33.183957 17645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:33.184298 17645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:33.186232 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "ac128f3e527445cf9ea5506452671031"
format_stamp: "Formatted at 2026-06-20 14:29:33 on dist-test-slave-pk3j"
server_key: "bdaf4b4eb2a33b7250cee78d2d700033"
server_key_iv: "629155b003bfd5f680691cbea376711e"
server_key_version: "encryptionkey@0"
I20260620 14:29:33.186780 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ac128f3e527445cf9ea5506452671031"
format_stamp: "Formatted at 2026-06-20 14:29:33 on dist-test-slave-pk3j"
server_key: "bdaf4b4eb2a33b7250cee78d2d700033"
server_key_iv: "629155b003bfd5f680691cbea376711e"
server_key_version: "encryptionkey@0"
I20260620 14:29:33.190274 17645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:29:33.192461 17660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:33.193638 17645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:33.193764 17645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "ac128f3e527445cf9ea5506452671031"
format_stamp: "Formatted at 2026-06-20 14:29:33 on dist-test-slave-pk3j"
server_key: "bdaf4b4eb2a33b7250cee78d2d700033"
server_key_iv: "629155b003bfd5f680691cbea376711e"
server_key_version: "encryptionkey@0"
I20260620 14:29:33.193864 17645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:33.209112 17645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:33.212273 17645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:33.212469 17645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:33.213001 17645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:33.213798 17645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:33.213865 17645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:33.213927 17645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:33.213976 17645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:33.222131 17645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45487
I20260620 14:29:33.222152 17773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45487 every 8 connection(s)
I20260620 14:29:33.222920 17645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:29:33.228641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 17645
I20260620 14:29:33.228804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:29:33.229085 392 external_mini_cluster.cc:1428] Setting key 97856164988911587ae4cda7075a2a19
Jun 20 14:29:33 dist-test-slave-pk3j krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:33.235733 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.224532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55551 (local address 127.0.98.62:45625)
0620 14:29:33.224658 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.224662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.225844 (+ 1182us) server_negotiation.cc:407] Connection header received
0620 14:29:33.226619 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.226622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.226666 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.226738 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:33.227753 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.228264 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.229518 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.229798 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.229909 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.232339 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:33.232343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:33.232345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:33.232375 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:33.233756 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:33.234260 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:33.234263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:33.234267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:33.234343 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:33.234691 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:33.234694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:33.234696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:33.234870 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:33.234979 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.235442 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.235568 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:29:33.236454 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.224870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45625 (local address 127.0.98.2:55551)
0620 14:29:33.225713 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:33.225751 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:33.226453 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:33.226880 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:33.226887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:33.227201 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:33.227635 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.227648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.228364 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.228366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.229394 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.229403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.229965 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.229969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.230144 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.230733 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:33.230748 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:33.232114 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:33.233874 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:33.233880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:33.233901 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:33.234147 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:33.234444 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:33.234448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:33.234450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:33.234586 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:33.234977 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:33.234984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:33.235210 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 14:29:33.235393 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 14:29:33.235683 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":107,"threads_started":1}
I20260620 14:29:33.237720 17774 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45625
I20260620 14:29:33.237942 17774 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:29:33.238379 17774 heartbeater.cc:511] Master 127.0.98.62:45625 requested a full tablet report, sending...
I20260620 14:29:33.239370 17449 ts_manager.cc:194] Registered new tserver with Master: ac128f3e527445cf9ea5506452671031 (127.0.98.2:45487)
I20260620 14:29:33.239920 17449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:29:33.253736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45625
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:33.358539 17779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:33.358754 17779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:33.358793 17779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:33.362599 17779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:33.362646 17779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:33.362699 17779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:29:33.367712 17779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:33.368541 17779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:33.369443 17779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:33.378283 17784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:33.378439 17779 server_base.cc:1034] running on GCE node
W20260620 14:29:33.378341 17787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:33.378283 17785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:33.378931 17779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:33.379469 17779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:33.380601 17779 hybrid_clock.cc:648] HybridClock initialized: now 1781965773380571 us; error 43 us; skew 500 ppm
Jun 20 14:29:33 dist-test-slave-pk3j krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:33.383250 17779 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:29:33.384083 17779 webserver.cc:460] Webserver started at http://127.0.98.3:40531/ using document root <none> and password file <none>
I20260620 14:29:33.384689 17779 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:33.384778 17779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:33.385017 17779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:33.386955 17779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "72553c1589fa4bb98c11d1ac5f553bdb"
format_stamp: "Formatted at 2026-06-20 14:29:33 on dist-test-slave-pk3j"
server_key: "c051b7cea4a02de0b56133f4eb4cfb79"
server_key_iv: "5573f2cf6d9ba96ee0e030f9ff22b96d"
server_key_version: "encryptionkey@0"
I20260620 14:29:33.387495 17779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "72553c1589fa4bb98c11d1ac5f553bdb"
format_stamp: "Formatted at 2026-06-20 14:29:33 on dist-test-slave-pk3j"
server_key: "c051b7cea4a02de0b56133f4eb4cfb79"
server_key_iv: "5573f2cf6d9ba96ee0e030f9ff22b96d"
server_key_version: "encryptionkey@0"
I20260620 14:29:33.391054 17779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:29:33.393296 17794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:33.394438 17779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:29:33.394582 17779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "72553c1589fa4bb98c11d1ac5f553bdb"
format_stamp: "Formatted at 2026-06-20 14:29:33 on dist-test-slave-pk3j"
server_key: "c051b7cea4a02de0b56133f4eb4cfb79"
server_key_iv: "5573f2cf6d9ba96ee0e030f9ff22b96d"
server_key_version: "encryptionkey@0"
I20260620 14:29:33.394682 17779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:33.406466 17779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:33.408946 17779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:33.409124 17779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:33.409670 17779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:33.410398 17779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:33.410466 17779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:33.410530 17779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:33.410578 17779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:33.419994 17779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45401
I20260620 14:29:33.420029 17907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45401 every 8 connection(s)
I20260620 14:29:33.420832 17779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:29:33.429055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 17779
I20260620 14:29:33.429177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
Jun 20 14:29:33 dist-test-slave-pk3j krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:33.429464 392 external_mini_cluster.cc:1428] Setting key ea7b9de48e8a07ca9f4b19dec166d153
I20260620 14:29:33.433280 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.422495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43839 (local address 127.0.98.62:45625)
0620 14:29:33.422618 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.422622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.423316 (+ 694us) server_negotiation.cc:407] Connection header received
0620 14:29:33.424189 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.424193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.424251 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.424351 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:33.425502 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.426014 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.427132 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.427349 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.427461 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.429761 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:33.429765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:33.429767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:33.429799 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:33.431216 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:33.431795 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:33.431800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:33.431802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:33.431893 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:33.432223 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:33.432227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:33.432229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:33.432407 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:33.432514 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.432981 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.433096 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:29:33.434001 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.422763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45625 (local address 127.0.98.3:43839)
0620 14:29:33.423175 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:33.423212 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:33.423992 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:33.424512 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:33.424518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:33.424865 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:33.425365 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.425380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.426133 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.426137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.426962 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.426971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.427495 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.427499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.427666 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.428210 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:33.428226 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:33.429613 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:33.431333 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:33.431339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:33.431355 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:33.431692 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:33.431990 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:33.431993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:33.431995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:33.432128 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:33.432512 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:33.432518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:33.432741 (+ 223us) client_negotiation.cc:769] Sending connection context
0620 14:29:33.432936 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 14:29:33.433218 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260620 14:29:33.434980 17908 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45625
I20260620 14:29:33.435170 17908 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:33.435556 17908 heartbeater.cc:511] Master 127.0.98.62:45625 requested a full tablet report, sending...
I20260620 14:29:33.436427 17449 ts_manager.cc:194] Registered new tserver with Master: 72553c1589fa4bb98c11d1ac5f553bdb (127.0.98.3:45401)
I20260620 14:29:33.436905 17449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43839
I20260620 14:29:33.443005 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:29:33.451190 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.444914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.0.98.62:45625)
0620 14:29:33.445022 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.445025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.445176 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:29:33.445328 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.445331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.445365 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.445434 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:33.446018 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.446475 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.447228 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.447423 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.447492 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.448705 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:33.448708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:33.448709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:33.448724 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:33.449917 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:33.450271 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:33.450274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:33.450275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:33.450339 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:33.450547 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:33.450549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:33.450550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:33.450740 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:33.450836 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.451010 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.451089 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:29:33.453817 17449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:29:33.456602 17449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:29:33.470417 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.465061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45067 (local address 127.0.0.1:34128)
0620 14:29:33.465723 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:33.465736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:33.465820 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:33.466998 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:33.467001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:33.467019 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:33.467110 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.467115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.468191 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.468194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.469066 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.469076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.469892 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.469895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.470010 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.470048 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:29:33.470155 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:29:33.470270 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260620 14:29:33.470620 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.464544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.0.98.2:45487)
0620 14:29:33.465142 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.465156 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.465181 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:29:33.465280 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.465285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.465504 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.466032 (+ 528us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:33.466198 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.467313 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.469158 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.469975 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.470118 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.470225 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.470373 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.470479 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":148,"threads_started":1}
I20260620 14:29:33.470652 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.464373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45487 (local address 127.0.0.1:50386)
0620 14:29:33.464727 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:33.464776 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:33.464904 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:33.465703 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:33.465710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:33.465743 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:33.465882 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.465889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.467490 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.467493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.469030 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.469039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.470147 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.470152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.470238 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.470257 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:29:33.470329 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:29:33.470385 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":126,"spinlock_wait_cycles":5120,"thread_start_us":78,"threads_started":1}
I20260620 14:29:33.471010 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.465116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.0.98.3:45401)
0620 14:29:33.465571 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.465583 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.465833 (+ 250us) server_negotiation.cc:407] Connection header received
0620 14:29:33.465924 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.465931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.466117 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.466266 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:33.466462 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.467312 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.469289 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.469991 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.470087 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.470214 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.470694 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.470904 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":107,"threads_started":1}
I20260620 14:29:33.471062 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.465278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.0.98.1:45067)
0620 14:29:33.466551 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.466567 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.466587 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:29:33.466658 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.466663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.466831 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.466992 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:33.467227 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.468056 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.469193 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.469767 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.469890 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.470029 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.470787 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.470893 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":80,"threads_started":1}
I20260620 14:29:33.472039 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.464767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45401 (local address 127.0.0.1:60020)
0620 14:29:33.465620 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:33.465635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:33.465744 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:33.466250 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:33.466252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:33.466266 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:33.466341 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.466345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.467429 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.467434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.469149 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.469160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.470282 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.470285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.470393 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.470415 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:29:33.470485 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:29:33.471913 (+ 1428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":326,"spinlock_wait_cycles":5248,"thread_start_us":91,"threads_started":1}
I20260620 14:29:33.473737 17574 tablet_service.cc:1467] Processing CreateTablet for tablet 676116a4fd5240f2afe321089b98b24b (DEFAULT_TABLE table=test-table [id=c17b1e2aec9f458bab48bbef7f91861f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:33.473752 17708 tablet_service.cc:1467] Processing CreateTablet for tablet 676116a4fd5240f2afe321089b98b24b (DEFAULT_TABLE table=test-table [id=c17b1e2aec9f458bab48bbef7f91861f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:33.474689 17574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676116a4fd5240f2afe321089b98b24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:33.474735 17708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676116a4fd5240f2afe321089b98b24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:33.474436 17842 tablet_service.cc:1467] Processing CreateTablet for tablet 676116a4fd5240f2afe321089b98b24b (DEFAULT_TABLE table=test-table [id=c17b1e2aec9f458bab48bbef7f91861f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:33.475287 17842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676116a4fd5240f2afe321089b98b24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:33.480508 17925 tablet_bootstrap.cc:492] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51: Bootstrap starting.
I20260620 14:29:33.482198 17926 tablet_bootstrap.cc:492] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: Bootstrap starting.
I20260620 14:29:33.482198 17927 tablet_bootstrap.cc:492] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: Bootstrap starting.
I20260620 14:29:33.483199 17925 tablet_bootstrap.cc:654] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:33.484195 17925 log.cc:824] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:33.484278 17926 tablet_bootstrap.cc:654] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:33.484278 17927 tablet_bootstrap.cc:654] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:33.485623 17927 log.cc:824] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:33.485625 17926 log.cc:824] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:33.486136 17925 tablet_bootstrap.cc:492] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51: No bootstrap required, opened a new log
I20260620 14:29:33.486335 17925 ts_tablet_manager.cc:1397] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:29:33.487474 17926 tablet_bootstrap.cc:492] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: No bootstrap required, opened a new log
I20260620 14:29:33.487670 17926 ts_tablet_manager.cc:1397] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:29:33.488301 17927 tablet_bootstrap.cc:492] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: No bootstrap required, opened a new log
I20260620 14:29:33.488472 17927 ts_tablet_manager.cc:1397] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:29:33.490322 17925 raft_consensus.cc:348] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } } peers { permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } } peers { permanent_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } }
I20260620 14:29:33.490581 17925 raft_consensus.cc:374] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:33.490636 17925 raft_consensus.cc:729] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8154ab6054f64745aee342fc12e91b51, State: Initialized, Role: FOLLOWER
I20260620 14:29:33.490763 17926 raft_consensus.cc:348] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } } peers { permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } } peers { permanent_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } }
I20260620 14:29:33.491051 17926 raft_consensus.cc:374] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:33.491112 17926 raft_consensus.cc:729] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac128f3e527445cf9ea5506452671031, State: Initialized, Role: FOLLOWER
I20260620 14:29:33.491104 17925 consensus_queue.cc:260] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } } peers { permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } } peers { permanent_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } }
I20260620 14:29:33.491731 17640 heartbeater.cc:503] Master 127.0.98.62:45625 was elected leader, sending a full tablet report...
I20260620 14:29:33.492077 17925 ts_tablet_manager.cc:1428] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:29:33.492161 17927 raft_consensus.cc:348] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } } peers { permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } } peers { permanent_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } }
I20260620 14:29:33.492394 17927 raft_consensus.cc:374] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:33.492475 17927 raft_consensus.cc:729] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72553c1589fa4bb98c11d1ac5f553bdb, State: Initialized, Role: FOLLOWER
I20260620 14:29:33.491518 17926 consensus_queue.cc:260] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } } peers { permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } } peers { permanent_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } }
I20260620 14:29:33.492942 17927 consensus_queue.cc:260] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } } peers { permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } } peers { permanent_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } }
I20260620 14:29:33.493470 17926 ts_tablet_manager.cc:1428] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260620 14:29:33.493834 17927 ts_tablet_manager.cc:1428] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 14:29:33.494175 17908 heartbeater.cc:503] Master 127.0.98.62:45625 was elected leader, sending a full tablet report...
I20260620 14:29:33.494390 17774 heartbeater.cc:503] Master 127.0.98.62:45625 was elected leader, sending a full tablet report...
W20260620 14:29:33.525818 17641 tablet.cc:2369] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:29:33.567411 17933 raft_consensus.cc:482] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:33.567575 17933 raft_consensus.cc:504] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } } peers { permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } } peers { permanent_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } }
I20260620 14:29:33.568557 17933 leader_election.cc:288] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac128f3e527445cf9ea5506452671031 (127.0.98.2:45487), 8154ab6054f64745aee342fc12e91b51 (127.0.98.1:45067)
I20260620 14:29:33.572388 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.568772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45487 (local address 127.0.98.3:59271)
0620 14:29:33.568889 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:33.568906 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:33.568997 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:33.569353 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:33.569358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:33.569376 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:33.569522 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.569528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.570353 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.570356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.571279 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.571288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.571951 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.571955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.572066 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.572125 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 14:29:33.572180 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:29:33.572253 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":80}
I20260620 14:29:33.572403 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.568908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59271 (local address 127.0.98.2:45487)
0620 14:29:33.569030 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.569035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.569071 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:29:33.569126 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.569129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.569179 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.569282 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:33.569630 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.570249 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.571400 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.571825 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.571934 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.571995 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.572216 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.572290 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:29:33.572443 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.569051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45067 (local address 127.0.98.3:36269)
0620 14:29:33.569375 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:33.569388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:33.569472 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:33.569711 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:33.569714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:33.569730 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:33.569840 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.569846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.570561 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.570563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.571364 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:33.571375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.571975 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.571978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:33.572139 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.572155 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:29:33.572250 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:29:33.572308 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":39,"thread_start_us":123,"threads_started":1}
I20260620 14:29:33.572489 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.569144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36269 (local address 127.0.98.1:45067)
0620 14:29:33.569331 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.569335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.569472 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:29:33.569546 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.569551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.569607 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.569678 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:33.569943 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.570454 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.571481 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.571835 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.571953 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.572007 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.572286 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.572351 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 14:29:33.573024 17594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676116a4fd5240f2afe321089b98b24b" candidate_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8154ab6054f64745aee342fc12e91b51" is_pre_election: true
I20260620 14:29:33.573076 17728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676116a4fd5240f2afe321089b98b24b" candidate_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac128f3e527445cf9ea5506452671031" is_pre_election: true
I20260620 14:29:33.573345 17594 raft_consensus.cc:2454] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72553c1589fa4bb98c11d1ac5f553bdb in term 0.
I20260620 14:29:33.573395 17728 raft_consensus.cc:2454] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72553c1589fa4bb98c11d1ac5f553bdb in term 0.
I20260620 14:29:33.573846 17796 leader_election.cc:302] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72553c1589fa4bb98c11d1ac5f553bdb, ac128f3e527445cf9ea5506452671031; no voters:
I20260620 14:29:33.574129 17933 raft_consensus.cc:2781] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:29:33.574213 17933 raft_consensus.cc:482] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:29:33.574256 17933 raft_consensus.cc:3037] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:33.575430 17933 raft_consensus.cc:504] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } } peers { permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } } peers { permanent_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } }
I20260620 14:29:33.575773 17933 leader_election.cc:288] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [CANDIDATE]: Term 1 election: Requested vote from peers ac128f3e527445cf9ea5506452671031 (127.0.98.2:45487), 8154ab6054f64745aee342fc12e91b51 (127.0.98.1:45067)
I20260620 14:29:33.576102 17728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676116a4fd5240f2afe321089b98b24b" candidate_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac128f3e527445cf9ea5506452671031"
I20260620 14:29:33.576159 17594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676116a4fd5240f2afe321089b98b24b" candidate_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8154ab6054f64745aee342fc12e91b51"
I20260620 14:29:33.576205 17728 raft_consensus.cc:3037] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:33.576303 17594 raft_consensus.cc:3037] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:33.577247 17728 raft_consensus.cc:2454] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72553c1589fa4bb98c11d1ac5f553bdb in term 1.
I20260620 14:29:33.577335 17594 raft_consensus.cc:2454] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72553c1589fa4bb98c11d1ac5f553bdb in term 1.
I20260620 14:29:33.577551 17796 leader_election.cc:302] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72553c1589fa4bb98c11d1ac5f553bdb, ac128f3e527445cf9ea5506452671031; no voters:
I20260620 14:29:33.577724 17933 raft_consensus.cc:2781] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:33.577941 17933 raft_consensus.cc:686] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 1 LEADER]: Becoming Leader. State: Replica: 72553c1589fa4bb98c11d1ac5f553bdb, State: Running, Role: LEADER
I20260620 14:29:33.578256 17933 consensus_queue.cc:237] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } } peers { permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } } peers { permanent_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } }
I20260620 14:29:33.581322 17447 catalog_manager.cc:5495] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 72553c1589fa4bb98c11d1ac5f553bdb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72553c1589fa4bb98c11d1ac5f553bdb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45401 } health_report { overall_health: HEALTHY } } }
I20260620 14:29:33.621043 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.617656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.0.98.3:45401)
0620 14:29:33.617798 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.617802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.617816 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:29:33.617939 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.617941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.617993 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.618099 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:29:33.618342 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.618818 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.619625 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.619842 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.619947 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.620245 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:29:33.620622 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:29:33.620755 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.620795 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.620847 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:29:33.626521 17594 raft_consensus.cc:1261] T 676116a4fd5240f2afe321089b98b24b P 8154ab6054f64745aee342fc12e91b51 [term 1 FOLLOWER]: Refusing update from remote peer 72553c1589fa4bb98c11d1ac5f553bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:33.626539 17728 raft_consensus.cc:1261] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031 [term 1 FOLLOWER]: Refusing update from remote peer 72553c1589fa4bb98c11d1ac5f553bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:33.627210 17935 consensus_queue.cc:1035] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac128f3e527445cf9ea5506452671031" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:33.627377 17933 consensus_queue.cc:1035] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8154ab6054f64745aee342fc12e91b51" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:33.636063 17940 mvcc.cc:205] Tried to move back new op lower bound from 7298931808767537152 to 7298931808578772992. Current Snapshot: MvccSnapshot[applied={T|T < 7298931808767537152}]
I20260620 14:29:33.643326 17447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:33.643534 17447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:33.645733 17447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4b013d7e01e4363b1d79a872cb178bb: Sending DeleteTablet for 3 replicas of tablet 676116a4fd5240f2afe321089b98b24b
I20260620 14:29:33.646462 17708 tablet_service.cc:1514] Processing DeleteTablet for tablet 676116a4fd5240f2afe321089b98b24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50386
I20260620 14:29:33.646462 17842 tablet_service.cc:1514] Processing DeleteTablet for tablet 676116a4fd5240f2afe321089b98b24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60020
I20260620 14:29:33.646917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 17511
I20260620 14:29:33.647024 17948 tablet_replica.cc:330] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: stopping tablet replica
I20260620 14:29:33.647282 17948 raft_consensus.cc:2229] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:33.647547 17949 tablet_replica.cc:330] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: stopping tablet replica
I20260620 14:29:33.647770 17949 raft_consensus.cc:2229] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:29:33.648068 17949 raft_consensus.cc:2258] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:33.648417 17948 raft_consensus.cc:2258] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:33.649719 17949 ts_tablet_manager.cc:1905] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:33.649714 17948 ts_tablet_manager.cc:1905] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:33.652403 17949 ts_tablet_manager.cc:1918] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:33.652560 17949 log.cc:1196] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/676116a4fd5240f2afe321089b98b24b
I20260620 14:29:33.652721 17948 ts_tablet_manager.cc:1918] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:33.652815 17948 log.cc:1196] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/676116a4fd5240f2afe321089b98b24b
I20260620 14:29:33.653115 17949 ts_tablet_manager.cc:1939] T 676116a4fd5240f2afe321089b98b24b P 72553c1589fa4bb98c11d1ac5f553bdb: Deleting consensus metadata
I20260620 14:29:33.653119 17948 ts_tablet_manager.cc:1939] T 676116a4fd5240f2afe321089b98b24b P ac128f3e527445cf9ea5506452671031: Deleting consensus metadata
I20260620 14:29:33.654156 17434 catalog_manager.cc:4841] TS ac128f3e527445cf9ea5506452671031 (127.0.98.2:45487): tablet 676116a4fd5240f2afe321089b98b24b (table test-table [id=c17b1e2aec9f458bab48bbef7f91861f]) successfully deleted
I20260620 14:29:33.654156 17436 catalog_manager.cc:4841] TS 72553c1589fa4bb98c11d1ac5f553bdb (127.0.98.3:45401): tablet 676116a4fd5240f2afe321089b98b24b (table test-table [id=c17b1e2aec9f458bab48bbef7f91861f]) successfully deleted
W20260620 14:29:33.656562 17434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45067 (error 108)
I20260620 14:29:33.656697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 17645
I20260620 14:29:33.658277 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.657890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45067 (local address 127.0.0.1:47512)
0620 14:29:33.658059 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:33.658169 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:29:33.658499 17434 catalog_manager.cc:4584] TS 8154ab6054f64745aee342fc12e91b51 (127.0.98.1:45067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 676116a4fd5240f2afe321089b98b24b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45067: connect: Connection refused (error 111)
I20260620 14:29:33.663065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 17779
I20260620 14:29:33.667769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 17417
2026-06-20T14:29:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2200 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:29:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:33Z Disabled control of system clock
I20260620 14:29:33.696389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36867
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:29:33.800717 17954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:33.800926 17954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:33.800959 17954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:33.804687 17954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:33.804731 17954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:33.804747 17954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:33.804761 17954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:33.804775 17954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:33.809278 17954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36867
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:33.810122 17954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:33.811390 17954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:33.819304 17960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:33.819422 17954 server_base.cc:1034] running on GCE node
W20260620 14:29:33.819314 17959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:33.819312 17962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:33.820076 17954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:33.820968 17954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:33.822132 17954 hybrid_clock.cc:648] HybridClock initialized: now 1781965773822113 us; error 30 us; skew 500 ppm
I20260620 14:29:33.823787 17954 webserver.cc:460] Webserver started at http://127.0.98.62:43257/ using document root <none> and password file <none>
I20260620 14:29:33.824313 17954 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:33.824371 17954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:33.824616 17954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:33.826864 17954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "c35cf3cb6305406086b5f013c03d03c7"
format_stamp: "Formatted at 2026-06-20 14:29:33 on dist-test-slave-pk3j"
server_key: "7576f3f3b4412d9a2934260df05cd23e"
server_key_iv: "db0dce79d8fb8735b43cae57ba590206"
server_key_version: "encryptionkey@0"
I20260620 14:29:33.827433 17954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "c35cf3cb6305406086b5f013c03d03c7"
format_stamp: "Formatted at 2026-06-20 14:29:33 on dist-test-slave-pk3j"
server_key: "7576f3f3b4412d9a2934260df05cd23e"
server_key_iv: "db0dce79d8fb8735b43cae57ba590206"
server_key_version: "encryptionkey@0"
I20260620 14:29:33.830792 17954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:29:33.832954 17968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:33.834117 17954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:33.834252 17954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "c35cf3cb6305406086b5f013c03d03c7"
format_stamp: "Formatted at 2026-06-20 14:29:33 on dist-test-slave-pk3j"
server_key: "7576f3f3b4412d9a2934260df05cd23e"
server_key_iv: "db0dce79d8fb8735b43cae57ba590206"
server_key_version: "encryptionkey@0"
I20260620 14:29:33.834355 17954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:33.851630 17954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:33.852198 17954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:33.852375 17954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:33.858788 17954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36867
I20260620 14:29:33.858799 18020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36867 every 8 connection(s)
I20260620 14:29:33.859618 17954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:33.861258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 17954
I20260620 14:29:33.861387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:33.861656 392 external_mini_cluster.cc:1428] Setting key 5f5cd9d99e6b07b0031e0c27da76f814
I20260620 14:29:33.862780 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:33.869419 18021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7: Bootstrap starting.
I20260620 14:29:33.871965 18021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:33.872012 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:33.863327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.0.98.62:36867)
0620 14:29:33.863997 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:33.864018 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:33.864060 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:29:33.864818 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:33.864840 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:33.865350 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:33.865698 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:33.866093 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.867511 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.868460 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:33.869107 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:33.869286 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:33.870110 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:33.870116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:33.870128 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:33.870585 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:33.870691 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:33.870718 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:33.870815 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:33.870918 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:29:33.871160 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":111,"threads_started":1}
I20260620 14:29:33.872807 18021 log.cc:824] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:33.874704 18021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7: No bootstrap required, opened a new log
I20260620 14:29:33.878078 18021 raft_consensus.cc:348] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35cf3cb6305406086b5f013c03d03c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36867 } }
I20260620 14:29:33.878271 18021 raft_consensus.cc:374] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:33.878355 18021 raft_consensus.cc:729] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35cf3cb6305406086b5f013c03d03c7, State: Initialized, Role: FOLLOWER
I20260620 14:29:33.878717 18021 consensus_queue.cc:260] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35cf3cb6305406086b5f013c03d03c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36867 } }
I20260620 14:29:33.878847 18021 raft_consensus.cc:388] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:33.878928 18021 raft_consensus.cc:482] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:33.879061 18021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:33.880013 18021 raft_consensus.cc:504] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35cf3cb6305406086b5f013c03d03c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36867 } }
I20260620 14:29:33.880324 18021 leader_election.cc:302] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c35cf3cb6305406086b5f013c03d03c7; no voters:
I20260620 14:29:33.880587 18021 leader_election.cc:288] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:33.880712 18026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:33.880927 18026 raft_consensus.cc:686] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [term 1 LEADER]: Becoming Leader. State: Replica: c35cf3cb6305406086b5f013c03d03c7, State: Running, Role: LEADER
I20260620 14:29:33.881196 18026 consensus_queue.cc:237] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35cf3cb6305406086b5f013c03d03c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36867 } }
I20260620 14:29:33.881721 18021 sys_catalog.cc:564] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:33.882884 18028 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c35cf3cb6305406086b5f013c03d03c7. Latest consensus state: current_term: 1 leader_uuid: "c35cf3cb6305406086b5f013c03d03c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35cf3cb6305406086b5f013c03d03c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36867 } } }
I20260620 14:29:33.882917 18027 sys_catalog.cc:455] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c35cf3cb6305406086b5f013c03d03c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c35cf3cb6305406086b5f013c03d03c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36867 } } }
I20260620 14:29:33.883015 18028 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:33.883021 18027 sys_catalog.cc:458] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:33.883402 18035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:33.886343 18035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:33.891220 18035 catalog_manager.cc:1347] Generated new cluster ID: e070ca3707094c0ab908a264f6ea1b44
I20260620 14:29:33.891286 18035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:33.897369 18035 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:33.898221 18035 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:33.902042 18035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7: Generated new TSK 0
I20260620 14:29:33.902586 18035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:33.909400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36867
--builtin_ntp_servers=127.0.98.20:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:29:34.017896 18045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:34.018127 18045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:34.018162 18045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:34.022071 18045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:34.022117 18045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:34.022203 18045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:29:34.026868 18045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:34.027860 18045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:34.029229 18045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:34.038298 18053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:34.038300 18051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:34.038363 18050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:34.038525 18045 server_base.cc:1034] running on GCE node
I20260620 14:29:34.038966 18045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:34.039533 18045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:34.040688 18045 hybrid_clock.cc:648] HybridClock initialized: now 1781965774040677 us; error 41 us; skew 500 ppm
I20260620 14:29:34.042538 18045 webserver.cc:460] Webserver started at http://127.0.98.1:35591/ using document root <none> and password file <none>
I20260620 14:29:34.043298 18045 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:34.043383 18045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:34.043634 18045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:34.045639 18045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "44645ceb224249c9908db3dbcc573fd9"
format_stamp: "Formatted at 2026-06-20 14:29:34 on dist-test-slave-pk3j"
server_key: "013a42179b1a0777140abc6021237eac"
server_key_iv: "92b072161b1585575ad9ccfc7a521e6a"
server_key_version: "encryptionkey@0"
I20260620 14:29:34.046178 18045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "44645ceb224249c9908db3dbcc573fd9"
format_stamp: "Formatted at 2026-06-20 14:29:34 on dist-test-slave-pk3j"
server_key: "013a42179b1a0777140abc6021237eac"
server_key_iv: "92b072161b1585575ad9ccfc7a521e6a"
server_key_version: "encryptionkey@0"
I20260620 14:29:34.049624 18045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:29:34.051831 18059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:34.052959 18045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:34.053089 18045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "44645ceb224249c9908db3dbcc573fd9"
format_stamp: "Formatted at 2026-06-20 14:29:34 on dist-test-slave-pk3j"
server_key: "013a42179b1a0777140abc6021237eac"
server_key_iv: "92b072161b1585575ad9ccfc7a521e6a"
server_key_version: "encryptionkey@0"
I20260620 14:29:34.053196 18045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:34.076107 18045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:34.076784 18045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:34.076982 18045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:34.077629 18045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:34.078516 18045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:34.078588 18045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:34.078655 18045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:34.078696 18045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:34.087755 18045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43541
I20260620 14:29:34.087797 18172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43541 every 8 connection(s)
I20260620 14:29:34.088624 18045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:29:34.095124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 18045
I20260620 14:29:34.095250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:29:34.095530 392 external_mini_cluster.cc:1428] Setting key 2b10683db1302d5d3e20964a0b095486
I20260620 14:29:34.097738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36867
--builtin_ntp_servers=127.0.98.20:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:29:34.098219 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.090427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54279 (local address 127.0.98.62:36867)
0620 14:29:34.090559 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:34.090562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:34.091590 (+ 1028us) server_negotiation.cc:407] Connection header received
0620 14:29:34.092921 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:34.092924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:34.092966 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:34.093038 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:34.093985 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.094505 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.096010 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.096260 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.096340 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.097578 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:34.097581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:34.097583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:34.097593 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:34.097612 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:34.097619 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:34.097678 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:34.097927 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:29:34.098005 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:29:34.099807 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.090832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36867 (local address 127.0.98.1:54279)
0620 14:29:34.091440 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:34.091479 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:34.092743 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:34.093238 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:34.093250 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:34.093370 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:34.093866 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.093880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.094611 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.094614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.095887 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.095905 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.096408 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.096412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.096593 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.097298 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:29:34.097300 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:34.097351 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:29:34.097425 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:34.097731 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:34.097740 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:34.097748 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:29:34.098381 (+ 633us) client_negotiation.cc:240] Negotiation successful
0620 14:29:34.098687 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":96,"threads_started":1}
I20260620 14:29:34.101117 18173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36867
I20260620 14:29:34.101361 18173 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:34.101892 18173 heartbeater.cc:511] Master 127.0.98.62:36867 requested a full tablet report, sending...
I20260620 14:29:34.103283 17985 ts_manager.cc:194] Registered new tserver with Master: 44645ceb224249c9908db3dbcc573fd9 (127.0.98.1:43541)
I20260620 14:29:34.104580 17985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54279
W20260620 14:29:34.202839 18176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:34.203095 18176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:34.203161 18176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:34.207110 18176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:34.207178 18176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:34.207261 18176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:29:34.211706 18176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:34.212646 18176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:34.213968 18176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:34.222851 18181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:34.222851 18184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:34.222891 18182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:34.223039 18176 server_base.cc:1034] running on GCE node
I20260620 14:29:34.223531 18176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:34.224126 18176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:34.225279 18176 hybrid_clock.cc:648] HybridClock initialized: now 1781965774225256 us; error 32 us; skew 500 ppm
I20260620 14:29:34.226919 18176 webserver.cc:460] Webserver started at http://127.0.98.2:44559/ using document root <none> and password file <none>
I20260620 14:29:34.227463 18176 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:34.227555 18176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:34.227763 18176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:34.229816 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "5b4f246c79ab43cf800c4938f9948a66"
format_stamp: "Formatted at 2026-06-20 14:29:34 on dist-test-slave-pk3j"
server_key: "441b7e6754980abaf33d6ab0a54aad1c"
server_key_iv: "58ab3d7aff239e12cee5767517ae3743"
server_key_version: "encryptionkey@0"
I20260620 14:29:34.230357 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5b4f246c79ab43cf800c4938f9948a66"
format_stamp: "Formatted at 2026-06-20 14:29:34 on dist-test-slave-pk3j"
server_key: "441b7e6754980abaf33d6ab0a54aad1c"
server_key_iv: "58ab3d7aff239e12cee5767517ae3743"
server_key_version: "encryptionkey@0"
I20260620 14:29:34.233966 18176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:29:34.236263 18190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:34.237589 18176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:29:34.237752 18176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "5b4f246c79ab43cf800c4938f9948a66"
format_stamp: "Formatted at 2026-06-20 14:29:34 on dist-test-slave-pk3j"
server_key: "441b7e6754980abaf33d6ab0a54aad1c"
server_key_iv: "58ab3d7aff239e12cee5767517ae3743"
server_key_version: "encryptionkey@0"
I20260620 14:29:34.237862 18176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:34.258025 18176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:34.258685 18176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:34.258876 18176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:34.259433 18176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:34.260362 18176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:34.260435 18176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:34.260505 18176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:34.260548 18176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:34.268996 18176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33751
I20260620 14:29:34.269027 18303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33751 every 8 connection(s)
I20260620 14:29:34.269891 18176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:29:34.273988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 18176
I20260620 14:29:34.274137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:29:34.274441 392 external_mini_cluster.cc:1428] Setting key 6e31544d7eb22090d917409a8f608736
I20260620 14:29:34.276288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36867
--builtin_ntp_servers=127.0.98.20:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:29:34.278895 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.271624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53953 (local address 127.0.98.62:36867)
0620 14:29:34.271763 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:34.271767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:34.272758 (+ 991us) server_negotiation.cc:407] Connection header received
0620 14:29:34.274167 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:34.274170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:34.274228 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:34.274338 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:34.275243 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.275748 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.276989 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.277227 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.277362 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.278429 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:34.278433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:34.278436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:34.278448 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:34.278473 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:34.278480 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:34.278554 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:34.278744 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 14:29:34.278793 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:29:34.280023 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.272018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36867 (local address 127.0.98.2:53953)
0620 14:29:34.272616 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:34.272652 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:34.273955 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:34.274530 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:34.274541 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:34.274653 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:34.275108 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.275120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.275908 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.275911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.276847 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.276856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.277397 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.277402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.277567 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.278170 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:29:34.278171 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:34.278208 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:29:34.278264 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:34.278567 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:34.278579 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:34.278588 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:29:34.278708 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:29:34.278989 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260620 14:29:34.281173 18304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36867
I20260620 14:29:34.281441 18304 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:34.282008 18304 heartbeater.cc:511] Master 127.0.98.62:36867 requested a full tablet report, sending...
I20260620 14:29:34.283100 17985 ts_manager.cc:194] Registered new tserver with Master: 5b4f246c79ab43cf800c4938f9948a66 (127.0.98.2:33751)
I20260620 14:29:34.283790 17985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53953
W20260620 14:29:34.383503 18307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:34.383754 18307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:34.383827 18307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:34.387835 18307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:34.387907 18307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:34.387986 18307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:29:34.393074 18307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:34.394073 18307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:34.395370 18307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:34.404446 18313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:34.404482 18312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:34.404511 18315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:34.404626 18307 server_base.cc:1034] running on GCE node
I20260620 14:29:34.405030 18307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:34.405661 18307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:34.406847 18307 hybrid_clock.cc:648] HybridClock initialized: now 1781965774406834 us; error 23 us; skew 500 ppm
I20260620 14:29:34.408574 18307 webserver.cc:460] Webserver started at http://127.0.98.3:35545/ using document root <none> and password file <none>
I20260620 14:29:34.409154 18307 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:34.409267 18307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:34.409503 18307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:34.411418 18307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "6596910c651f42b39000c2c4ee3859f1"
format_stamp: "Formatted at 2026-06-20 14:29:34 on dist-test-slave-pk3j"
server_key: "f4053761af62977870ebe849efa91bae"
server_key_iv: "b810816357495322ec422910eadc1ab3"
server_key_version: "encryptionkey@0"
I20260620 14:29:34.411918 18307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6596910c651f42b39000c2c4ee3859f1"
format_stamp: "Formatted at 2026-06-20 14:29:34 on dist-test-slave-pk3j"
server_key: "f4053761af62977870ebe849efa91bae"
server_key_iv: "b810816357495322ec422910eadc1ab3"
server_key_version: "encryptionkey@0"
I20260620 14:29:34.415401 18307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:29:34.417675 18321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:34.418838 18307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:29:34.418928 18307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "6596910c651f42b39000c2c4ee3859f1"
format_stamp: "Formatted at 2026-06-20 14:29:34 on dist-test-slave-pk3j"
server_key: "f4053761af62977870ebe849efa91bae"
server_key_iv: "b810816357495322ec422910eadc1ab3"
server_key_version: "encryptionkey@0"
I20260620 14:29:34.419006 18307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:34.432331 18307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:34.432945 18307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:34.433122 18307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:34.433691 18307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:34.434562 18307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:34.434608 18307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:34.434675 18307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:34.434715 18307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:34.443639 18307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34233
I20260620 14:29:34.443660 18434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34233 every 8 connection(s)
I20260620 14:29:34.444481 18307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:29:34.452293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 18307
I20260620 14:29:34.452412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:29:34.452708 392 external_mini_cluster.cc:1428] Setting key de2f1d4b8548bd525ac1c263c5833184
I20260620 14:29:34.452801 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.446193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46875 (local address 127.0.98.62:36867)
0620 14:29:34.446320 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:34.446323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:34.447199 (+ 876us) server_negotiation.cc:407] Connection header received
0620 14:29:34.448470 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:34.448473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:34.448517 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:34.448587 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:34.449379 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.449906 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.450950 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.451151 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.451224 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.452298 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:34.452301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:34.452302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:34.452312 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:34.452326 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:34.452332 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:34.452416 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:34.452591 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:29:34.452638 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:29:34.453814 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.446500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36867 (local address 127.0.98.3:46875)
0620 14:29:34.447083 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:34.447119 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:34.448282 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:34.448705 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:34.448711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:34.448786 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:34.449246 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.449258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.450003 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.450005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.450849 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.450858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.451265 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.451268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.451404 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.452019 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:29:34.452020 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:34.452086 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:29:34.452162 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:34.452436 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:34.452446 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:34.452453 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:29:34.452576 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:29:34.452858 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":96,"threads_started":1}
I20260620 14:29:34.454936 18435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36867
I20260620 14:29:34.455161 18435 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:34.455621 18435 heartbeater.cc:511] Master 127.0.98.62:36867 requested a full tablet report, sending...
I20260620 14:29:34.456638 17985 ts_manager.cc:194] Registered new tserver with Master: 6596910c651f42b39000c2c4ee3859f1 (127.0.98.3:34233)
I20260620 14:29:34.457158 17985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46875
I20260620 14:29:34.466383 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:29:34.471809 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.468239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.0.98.62:36867)
0620 14:29:34.468387 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:34.468393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:34.468529 (+ 136us) server_negotiation.cc:407] Connection header received
0620 14:29:34.468659 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:34.468662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:34.468712 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:34.468793 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:34.469054 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.469546 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.470364 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.470568 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.470684 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.471410 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:34.471414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:34.471416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:34.471435 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:34.471450 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:34.471458 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:34.471531 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:34.471623 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:29:34.471668 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:29:34.474274 17985 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:29:34.477151 17985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:29:34.490996 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.485649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34233 (local address 127.0.0.1:35100)
0620 14:29:34.485933 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:34.485947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:34.486024 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:34.487152 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:34.487157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:34.487176 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:34.487264 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.487270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.488538 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.488543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.489713 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.489722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.490461 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.490464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.490560 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.490593 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:29:34.490728 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 14:29:34.490845 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":23,"thread_start_us":108,"threads_started":1}
I20260620 14:29:34.491008 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.486138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.0.98.3:34233)
0620 14:29:34.486541 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:34.486586 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:34.486607 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:29:34.486668 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:34.486672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:34.486846 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:34.487000 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:34.487393 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.488436 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.489826 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.490341 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.490460 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.490639 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:34.490764 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:29:34.490855 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
I20260620 14:29:34.491479 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.485744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.0.98.2:33751)
0620 14:29:34.486553 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:34.486564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:34.486589 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:29:34.486664 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:34.486669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:34.486846 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:34.487000 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:34.487207 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.487980 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.489877 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.490629 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.490775 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.490959 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:34.491240 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 14:29:34.491333 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":116,"threads_started":1}
I20260620 14:29:34.492043 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.484911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43541 (local address 127.0.0.1:56646)
0620 14:29:34.485414 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:34.485458 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:34.485629 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:34.486558 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:34.486563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:34.486587 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:34.486719 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.486724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.488231 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.488235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.489734 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.489745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.490798 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.490801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.490886 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.490910 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:29:34.490961 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:29:34.491032 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":97,"spinlock_wait_cycles":7552,"thread_start_us":81,"threads_started":1}
I20260620 14:29:34.492043 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.485018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33751 (local address 127.0.0.1:43470)
0620 14:29:34.485421 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:34.485463 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:34.485627 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:34.486989 (+ 1362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:34.486991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:34.487011 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:34.487094 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.487099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.488096 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.488102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.489734 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.489745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.491013 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.491016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.491097 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.491126 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:29:34.491207 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:29:34.491275 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":198,"spinlock_wait_cycles":6272,"thread_start_us":105,"threads_started":1}
I20260620 14:29:34.492811 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.485048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.0.98.1:43541)
0620 14:29:34.485918 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:34.485929 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:34.485957 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:29:34.486036 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:34.486043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:34.486275 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:34.486842 (+ 567us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:34.486978 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.488122 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.489876 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.490629 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.492335 (+ 1706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.492511 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:34.492553 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:29:34.492633 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":84,"threads_started":1}
I20260620 14:29:34.494093 18369 tablet_service.cc:1467] Processing CreateTablet for tablet b7269c46c3434892a9aa59548ece4309 (DEFAULT_TABLE table=test-table [id=3bf41842426248738163eba682fe6975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:34.494714 18238 tablet_service.cc:1467] Processing CreateTablet for tablet b7269c46c3434892a9aa59548ece4309 (DEFAULT_TABLE table=test-table [id=3bf41842426248738163eba682fe6975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:34.495265 18369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7269c46c3434892a9aa59548ece4309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:34.494912 18107 tablet_service.cc:1467] Processing CreateTablet for tablet b7269c46c3434892a9aa59548ece4309 (DEFAULT_TABLE table=test-table [id=3bf41842426248738163eba682fe6975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:34.495661 18238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7269c46c3434892a9aa59548ece4309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:34.495815 18107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7269c46c3434892a9aa59548ece4309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:34.501283 18452 tablet_bootstrap.cc:492] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9: Bootstrap starting.
I20260620 14:29:34.502735 18453 tablet_bootstrap.cc:492] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: Bootstrap starting.
I20260620 14:29:34.503739 18452 tablet_bootstrap.cc:654] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:34.504456 18452 log.cc:824] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:34.504514 18453 tablet_bootstrap.cc:654] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:34.505148 18453 log.cc:824] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:34.505573 18454 tablet_bootstrap.cc:492] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: Bootstrap starting.
I20260620 14:29:34.506971 18452 tablet_bootstrap.cc:492] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9: No bootstrap required, opened a new log
I20260620 14:29:34.507077 18453 tablet_bootstrap.cc:492] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: No bootstrap required, opened a new log
I20260620 14:29:34.507162 18452 ts_tablet_manager.cc:1397] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:29:34.507230 18453 ts_tablet_manager.cc:1397] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:29:34.507488 18454 tablet_bootstrap.cc:654] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:34.508105 18454 log.cc:824] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:34.509613 18454 tablet_bootstrap.cc:492] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: No bootstrap required, opened a new log
I20260620 14:29:34.509796 18454 ts_tablet_manager.cc:1397] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 14:29:34.511137 18453 raft_consensus.cc:348] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 } } peers { permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } } peers { permanent_uuid: "6596910c651f42b39000c2c4ee3859f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34233 } }
I20260620 14:29:34.511137 18452 raft_consensus.cc:348] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 } } peers { permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } } peers { permanent_uuid: "6596910c651f42b39000c2c4ee3859f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34233 } }
I20260620 14:29:34.511394 18452 raft_consensus.cc:374] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:34.511394 18453 raft_consensus.cc:374] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:34.511443 18453 raft_consensus.cc:729] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6596910c651f42b39000c2c4ee3859f1, State: Initialized, Role: FOLLOWER
I20260620 14:29:34.511451 18452 raft_consensus.cc:729] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44645ceb224249c9908db3dbcc573fd9, State: Initialized, Role: FOLLOWER
I20260620 14:29:34.511933 18452 consensus_queue.cc:260] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 } } peers { permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } } peers { permanent_uuid: "6596910c651f42b39000c2c4ee3859f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34233 } }
I20260620 14:29:34.511933 18453 consensus_queue.cc:260] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 } } peers { permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } } peers { permanent_uuid: "6596910c651f42b39000c2c4ee3859f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34233 } }
I20260620 14:29:34.512874 18452 ts_tablet_manager.cc:1428] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260620 14:29:34.512868 18435 heartbeater.cc:503] Master 127.0.98.62:36867 was elected leader, sending a full tablet report...
I20260620 14:29:34.512874 18453 ts_tablet_manager.cc:1428] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:29:34.513187 18173 heartbeater.cc:503] Master 127.0.98.62:36867 was elected leader, sending a full tablet report...
I20260620 14:29:34.513314 18454 raft_consensus.cc:348] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 } } peers { permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } } peers { permanent_uuid: "6596910c651f42b39000c2c4ee3859f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34233 } }
I20260620 14:29:34.513528 18454 raft_consensus.cc:374] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:34.513581 18454 raft_consensus.cc:729] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4f246c79ab43cf800c4938f9948a66, State: Initialized, Role: FOLLOWER
I20260620 14:29:34.513971 18454 consensus_queue.cc:260] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 } } peers { permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } } peers { permanent_uuid: "6596910c651f42b39000c2c4ee3859f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34233 } }
I20260620 14:29:34.514597 18304 heartbeater.cc:503] Master 127.0.98.62:36867 was elected leader, sending a full tablet report...
I20260620 14:29:34.515682 18454 ts_tablet_manager.cc:1428] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 14:29:34.520710 18305 tablet.cc:2369] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:29:34.586534 18459 raft_consensus.cc:482] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:34.586833 18459 raft_consensus.cc:504] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 } } peers { permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } } peers { permanent_uuid: "6596910c651f42b39000c2c4ee3859f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34233 } }
I20260620 14:29:34.588047 18459 leader_election.cc:288] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44645ceb224249c9908db3dbcc573fd9 (127.0.98.1:43541), 5b4f246c79ab43cf800c4938f9948a66 (127.0.98.2:33751)
W20260620 14:29:34.589797 18174 tablet.cc:2369] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:29:34.592674 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.588425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38085 (local address 127.0.98.1:43541)
0620 14:29:34.588556 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:34.588560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:34.588578 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:29:34.588731 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:34.588734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:34.588813 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:34.588904 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:34.589261 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.589909 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.591363 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.591970 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.592091 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.592158 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:34.592478 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 14:29:34.592554 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:29:34.592674 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.588493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52317 (local address 127.0.98.2:33751)
0620 14:29:34.588614 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:34.588619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:34.588954 (+ 335us) server_negotiation.cc:407] Connection header received
0620 14:29:34.589068 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:34.589074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:34.589129 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:34.589237 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:34.589491 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.590170 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.591482 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.592046 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.592133 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.592185 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:34.592478 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 14:29:34.592548 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:29:34.592697 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.588290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43541 (local address 127.0.98.3:38085)
0620 14:29:34.588510 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:34.588526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:34.588622 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:34.588946 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:34.588949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:34.588971 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:34.589114 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.589123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.590040 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.590044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.591227 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.591237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.592120 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.592125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.592313 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.592368 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:29:34.592445 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:29:34.592545 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":81}
I20260620 14:29:34.592712 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.588290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33751 (local address 127.0.98.3:52317)
0620 14:29:34.588851 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:34.588868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:34.588963 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:34.589248 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:34.589251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:34.589268 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:34.589379 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.589386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.590284 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.590288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.591371 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:34.591378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.592142 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.592145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:34.592332 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.592372 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:29:34.592445 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:29:34.592545 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":229,"thread_start_us":143,"threads_started":1}
I20260620 14:29:34.593520 18258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7269c46c3434892a9aa59548ece4309" candidate_uuid: "6596910c651f42b39000c2c4ee3859f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b4f246c79ab43cf800c4938f9948a66" is_pre_election: true
I20260620 14:29:34.593518 18127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7269c46c3434892a9aa59548ece4309" candidate_uuid: "6596910c651f42b39000c2c4ee3859f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44645ceb224249c9908db3dbcc573fd9" is_pre_election: true
I20260620 14:29:34.593834 18127 raft_consensus.cc:2454] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6596910c651f42b39000c2c4ee3859f1 in term 0.
I20260620 14:29:34.593834 18258 raft_consensus.cc:2454] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6596910c651f42b39000c2c4ee3859f1 in term 0.
I20260620 14:29:34.594336 18324 leader_election.cc:302] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44645ceb224249c9908db3dbcc573fd9, 6596910c651f42b39000c2c4ee3859f1; no voters:
I20260620 14:29:34.594595 18459 raft_consensus.cc:2781] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:29:34.594695 18459 raft_consensus.cc:482] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:29:34.594738 18459 raft_consensus.cc:3037] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:34.595706 18459 raft_consensus.cc:504] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 } } peers { permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } } peers { permanent_uuid: "6596910c651f42b39000c2c4ee3859f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34233 } }
I20260620 14:29:34.596057 18459 leader_election.cc:288] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [CANDIDATE]: Term 1 election: Requested vote from peers 44645ceb224249c9908db3dbcc573fd9 (127.0.98.1:43541), 5b4f246c79ab43cf800c4938f9948a66 (127.0.98.2:33751)
I20260620 14:29:34.596349 18127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7269c46c3434892a9aa59548ece4309" candidate_uuid: "6596910c651f42b39000c2c4ee3859f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44645ceb224249c9908db3dbcc573fd9"
I20260620 14:29:34.596483 18258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7269c46c3434892a9aa59548ece4309" candidate_uuid: "6596910c651f42b39000c2c4ee3859f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b4f246c79ab43cf800c4938f9948a66"
I20260620 14:29:34.596470 18127 raft_consensus.cc:3037] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:34.596653 18258 raft_consensus.cc:3037] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:34.597640 18127 raft_consensus.cc:2454] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6596910c651f42b39000c2c4ee3859f1 in term 1.
I20260620 14:29:34.597826 18258 raft_consensus.cc:2454] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6596910c651f42b39000c2c4ee3859f1 in term 1.
I20260620 14:29:34.597967 18324 leader_election.cc:302] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44645ceb224249c9908db3dbcc573fd9, 6596910c651f42b39000c2c4ee3859f1; no voters:
I20260620 14:29:34.598158 18459 raft_consensus.cc:2781] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:34.598380 18459 raft_consensus.cc:686] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 1 LEADER]: Becoming Leader. State: Replica: 6596910c651f42b39000c2c4ee3859f1, State: Running, Role: LEADER
I20260620 14:29:34.598711 18459 consensus_queue.cc:237] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 } } peers { permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } } peers { permanent_uuid: "6596910c651f42b39000c2c4ee3859f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34233 } }
I20260620 14:29:34.602010 17983 catalog_manager.cc:5495] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6596910c651f42b39000c2c4ee3859f1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6596910c651f42b39000c2c4ee3859f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6596910c651f42b39000c2c4ee3859f1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34233 } health_report { overall_health: HEALTHY } } }
I20260620 14:29:34.616858 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.613626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.0.98.3:34233)
0620 14:29:34.613744 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:34.613747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:34.613760 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:29:34.613860 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:34.613863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:34.613914 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:34.613997 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:29:34.614184 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.614678 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.615484 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:34.615685 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:34.615757 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:34.616003 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:29:34.616403 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:29:34.616489 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:34.616659 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:29:34.616705 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:29:34.622486 18127 raft_consensus.cc:1261] T b7269c46c3434892a9aa59548ece4309 P 44645ceb224249c9908db3dbcc573fd9 [term 1 FOLLOWER]: Refusing update from remote peer 6596910c651f42b39000c2c4ee3859f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:34.622537 18258 raft_consensus.cc:1261] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66 [term 1 FOLLOWER]: Refusing update from remote peer 6596910c651f42b39000c2c4ee3859f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:34.623196 18462 consensus_queue.cc:1035] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b4f246c79ab43cf800c4938f9948a66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:34.623415 18459 consensus_queue.cc:1035] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44645ceb224249c9908db3dbcc573fd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:34.631079 18466 mvcc.cc:205] Tried to move back new op lower bound from 7298931812846333952 to 7298931812759015424. Current Snapshot: MvccSnapshot[applied={T|T < 7298931812846333952}]
I20260620 14:29:34.640014 17983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:34.640156 17983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:34.642500 17983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c35cf3cb6305406086b5f013c03d03c7: Sending DeleteTablet for 3 replicas of tablet b7269c46c3434892a9aa59548ece4309
I20260620 14:29:34.643211 18238 tablet_service.cc:1514] Processing DeleteTablet for tablet b7269c46c3434892a9aa59548ece4309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:34 UTC) from {username='slave'} at 127.0.0.1:43470
I20260620 14:29:34.643317 18107 tablet_service.cc:1514] Processing DeleteTablet for tablet b7269c46c3434892a9aa59548ece4309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:34 UTC) from {username='slave'} at 127.0.0.1:56646
I20260620 14:29:34.643349 18369 tablet_service.cc:1514] Processing DeleteTablet for tablet b7269c46c3434892a9aa59548ece4309 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:34 UTC) from {username='slave'} at 127.0.0.1:35100
I20260620 14:29:34.643726 18475 tablet_replica.cc:330] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: stopping tablet replica
I20260620 14:29:34.644040 18475 raft_consensus.cc:2229] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:34.644316 18477 tablet_replica.cc:330] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: stopping tablet replica
I20260620 14:29:34.644345 18475 raft_consensus.cc:2258] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:34.644436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 18045
I20260620 14:29:34.644608 18477 raft_consensus.cc:2229] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:29:34.644994 18477 raft_consensus.cc:2258] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:34.646767 18477 ts_tablet_manager.cc:1905] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:34.646767 18475 ts_tablet_manager.cc:1905] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:34.649405 18475 ts_tablet_manager.cc:1918] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:34.649412 18477 ts_tablet_manager.cc:1918] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:34.649504 18475 log.cc:1196] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/b7269c46c3434892a9aa59548ece4309
I20260620 14:29:34.649504 18477 log.cc:1196] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/b7269c46c3434892a9aa59548ece4309
I20260620 14:29:34.649852 18475 ts_tablet_manager.cc:1939] T b7269c46c3434892a9aa59548ece4309 P 5b4f246c79ab43cf800c4938f9948a66: Deleting consensus metadata
I20260620 14:29:34.649860 18477 ts_tablet_manager.cc:1939] T b7269c46c3434892a9aa59548ece4309 P 6596910c651f42b39000c2c4ee3859f1: Deleting consensus metadata
I20260620 14:29:34.651000 17972 catalog_manager.cc:4841] TS 6596910c651f42b39000c2c4ee3859f1 (127.0.98.3:34233): tablet b7269c46c3434892a9aa59548ece4309 (table test-table [id=3bf41842426248738163eba682fe6975]) successfully deleted
I20260620 14:29:34.651000 17970 catalog_manager.cc:4841] TS 5b4f246c79ab43cf800c4938f9948a66 (127.0.98.2:33751): tablet b7269c46c3434892a9aa59548ece4309 (table test-table [id=3bf41842426248738163eba682fe6975]) successfully deleted
W20260620 14:29:34.654856 17971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43541 (error 108)
I20260620 14:29:34.655639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 18176
I20260620 14:29:34.655844 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:34.655560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43541 (local address 127.0.0.1:56654)
0620 14:29:34.655695 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:34.655760 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:29:34.657588 17971 catalog_manager.cc:4584] TS 44645ceb224249c9908db3dbcc573fd9 (127.0.98.1:43541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b7269c46c3434892a9aa59548ece4309: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43541: connect: Connection refused (error 111)
I20260620 14:29:34.661842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 18307
I20260620 14:29:34.666678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 17954
2026-06-20T14:29:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (998 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:34 dist-test-slave-pk3j krb5kdc[18481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:34 dist-test-slave-pk3j krb5kdc[18481](info): set up 2 sockets
Jun 20 14:29:34 dist-test-slave-pk3j krb5kdc[18481](info): commencing operation
krb5kdc: starting...
W20260620 14:29:35.697088 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:35 dist-test-slave-pk3j krb5kdc[18481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:29:35.806290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33107
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:35.913127 18494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:35.913426 18494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:35.913483 18494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:35.917411 18494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:35.917456 18494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:35.917475 18494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:35.917490 18494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:35.917526 18494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:35.922459 18494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:35.923388 18494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:35.924360 18494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:35.932279 18500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:35.932276 18499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:35.932294 18502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:35.933009 18494 server_base.cc:1034] running on GCE node
I20260620 14:29:35.933602 18494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:35.934629 18494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:35.935808 18494 hybrid_clock.cc:648] HybridClock initialized: now 1781965775935778 us; error 50 us; skew 500 ppm
Jun 20 14:29:35 dist-test-slave-pk3j krb5kdc[18481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:35.938444 18494 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:29:35.939482 18494 webserver.cc:460] Webserver started at http://127.0.98.62:35289/ using document root <none> and password file <none>
I20260620 14:29:35.940181 18494 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:35.940256 18494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:35.940486 18494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:35.943063 18494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb"
format_stamp: "Formatted at 2026-06-20 14:29:35 on dist-test-slave-pk3j"
server_key: "f2b698bdcc37246f74ea77abbfd37ee6"
server_key_iv: "67e8d5cceafbb77dfdccc889b59e8aad"
server_key_version: "encryptionkey@0"
I20260620 14:29:35.943753 18494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb"
format_stamp: "Formatted at 2026-06-20 14:29:35 on dist-test-slave-pk3j"
server_key: "f2b698bdcc37246f74ea77abbfd37ee6"
server_key_iv: "67e8d5cceafbb77dfdccc889b59e8aad"
server_key_version: "encryptionkey@0"
I20260620 14:29:35.948424 18494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:29:35.952005 18509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:35.953486 18494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:29:35.953611 18494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb"
format_stamp: "Formatted at 2026-06-20 14:29:35 on dist-test-slave-pk3j"
server_key: "f2b698bdcc37246f74ea77abbfd37ee6"
server_key_iv: "67e8d5cceafbb77dfdccc889b59e8aad"
server_key_version: "encryptionkey@0"
I20260620 14:29:35.953733 18494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:35.962707 18494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:35.965373 18494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:35.965550 18494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:35.972062 18494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33107
I20260620 14:29:35.972074 18561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33107 every 8 connection(s)
I20260620 14:29:35.972952 18494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:35.974051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 18494
I20260620 14:29:35.974200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:35.974498 392 external_mini_cluster.cc:1428] Setting key d89cb297e61d0e455ec05d8195f954cc
I20260620 14:29:35.975946 18562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:35.982131 18562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb: Bootstrap starting.
Jun 20 14:29:35 dist-test-slave-pk3j krb5kdc[18481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:35.984354 18562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:35.985036 18562 log.cc:824] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:35.986824 18562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb: No bootstrap required, opened a new log
I20260620 14:29:35.988242 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:35.975761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.0.98.62:33107)
0620 14:29:35.976162 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:35.976177 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:35.976211 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:29:35.976821 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:35.976836 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:35.977157 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:35.977487 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:35.978354 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:35.979285 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:35.980408 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:35.980830 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:35.980945 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:35.983625 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:35.983629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:35.983641 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:35.983667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:35.985420 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:35.985841 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:35.985851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:35.985855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:35.985961 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:35.986233 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:35.986237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:35.986239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:35.986643 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:35.986839 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:35.987110 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:29:35.987396 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":153,"threads_started":1}
I20260620 14:29:35.990598 18562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } }
I20260620 14:29:35.990813 18562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:35.990849 18562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a3379517c384d2d8f8d7e4a5d3f4aeb, State: Initialized, Role: FOLLOWER
I20260620 14:29:35.991173 18562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } }
I20260620 14:29:35.991273 18562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:35.991315 18562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:35.991387 18562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:35.992128 18562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } }
I20260620 14:29:35.992393 18562 leader_election.cc:302] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a3379517c384d2d8f8d7e4a5d3f4aeb; no voters:
I20260620 14:29:35.992614 18562 leader_election.cc:288] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:35.992727 18567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:35.992985 18567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [term 1 LEADER]: Becoming Leader. State: Replica: 1a3379517c384d2d8f8d7e4a5d3f4aeb, State: Running, Role: LEADER
I20260620 14:29:35.993305 18567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } }
I20260620 14:29:35.993752 18562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:35.995002 18568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } } }
I20260620 14:29:35.995208 18568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:35.995177 18569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a3379517c384d2d8f8d7e4a5d3f4aeb. Latest consensus state: current_term: 1 leader_uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3379517c384d2d8f8d7e4a5d3f4aeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33107 } } }
I20260620 14:29:35.995308 18569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:35.995744 18581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:35.998950 18581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:36.004302 18581 catalog_manager.cc:1347] Generated new cluster ID: c04338b6064040768bc52433677534c3
I20260620 14:29:36.004362 18581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:36.042124 18581 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:36.043395 18581 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:36.055636 18581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb: Generated new TSK 0
I20260620 14:29:36.056484 18581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:29:36.090217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:36.196734 18588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:36.197067 18588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:36.197113 18588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:36.200912 18588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:36.200958 18588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:36.201010 18588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:29:36.205904 18588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:36.206692 18588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:36.207547 18588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:36.216162 18593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:36.216176 18596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:36.216360 18588 server_base.cc:1034] running on GCE node
W20260620 14:29:36.216460 18594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:36.216873 18588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:36.217492 18588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:36.218665 18588 hybrid_clock.cc:648] HybridClock initialized: now 1781965776218642 us; error 40 us; skew 500 ppm
Jun 20 14:29:36 dist-test-slave-pk3j krb5kdc[18481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:36.221295 18588 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:29:36.221979 18588 webserver.cc:460] Webserver started at http://127.0.98.1:35195/ using document root <none> and password file <none>
I20260620 14:29:36.222429 18588 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:36.222481 18588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:36.222620 18588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:36.224354 18588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "d20e2fe413ea415a8bc10c379ae58d2e"
format_stamp: "Formatted at 2026-06-20 14:29:36 on dist-test-slave-pk3j"
server_key: "381dd2d3d8c74bdb54c2c07d53c3679f"
server_key_iv: "3c68934f8b5526178c0b2812656fb383"
server_key_version: "encryptionkey@0"
I20260620 14:29:36.224822 18588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d20e2fe413ea415a8bc10c379ae58d2e"
format_stamp: "Formatted at 2026-06-20 14:29:36 on dist-test-slave-pk3j"
server_key: "381dd2d3d8c74bdb54c2c07d53c3679f"
server_key_iv: "3c68934f8b5526178c0b2812656fb383"
server_key_version: "encryptionkey@0"
I20260620 14:29:36.228072 18588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:29:36.230274 18603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:36.231412 18588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:29:36.231544 18588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "d20e2fe413ea415a8bc10c379ae58d2e"
format_stamp: "Formatted at 2026-06-20 14:29:36 on dist-test-slave-pk3j"
server_key: "381dd2d3d8c74bdb54c2c07d53c3679f"
server_key_iv: "3c68934f8b5526178c0b2812656fb383"
server_key_version: "encryptionkey@0"
I20260620 14:29:36.231648 18588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:36.251255 18588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:36.253834 18588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:36.254019 18588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:36.254560 18588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:36.255318 18588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:36.255393 18588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:36.255476 18588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:36.255510 18588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:36.263899 18588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46687
I20260620 14:29:36.263931 18716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46687 every 8 connection(s)
I20260620 14:29:36.264719 18588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:29:36.265594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 18588
I20260620 14:29:36.265724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:29:36.266049 392 external_mini_cluster.cc:1428] Setting key 1237f8f9f2ed61f17ee8ea5779e94db5
Jun 20 14:29:36 dist-test-slave-pk3j krb5kdc[18481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:29:36.279713 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.266571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50443 (local address 127.0.98.62:33107)
0620 14:29:36.266690 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:36.266695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:36.268053 (+ 1358us) server_negotiation.cc:407] Connection header received
0620 14:29:36.269395 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:36.269399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:36.269453 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:36.269526 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:36.270883 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.271487 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.273318 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.273532 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.273630 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.276262 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:36.276266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:36.276268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:36.276291 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:36.277688 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:36.278170 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:36.278176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:36.278179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:36.278280 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:36.278625 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:36.278632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:36.278636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:36.278911 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:36.279019 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:36.279476 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 14:29:36.279598 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:29:36.280759 18719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.266975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33107 (local address 127.0.98.1:50443)
0620 14:29:36.267895 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:36.267947 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:36.269114 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:36.269689 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:36.269696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:36.270066 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:36.270711 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.270750 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.271627 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.271630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.273175 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.273188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.273704 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.273708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.273926 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.274511 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:36.274534 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:36.276094 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:36.277804 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:36.277810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:36.277824 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:36.278061 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:36.278390 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:36.278393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:36.278394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:36.278523 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:36.279010 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:36.279014 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:36.279216 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 14:29:36.279418 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 14:29:36.279763 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":144,"threads_started":1}
I20260620 14:29:36.282063 18717 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33107
I20260620 14:29:36.282279 18717 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:36.282713 18717 heartbeater.cc:511] Master 127.0.98.62:33107 requested a full tablet report, sending...
I20260620 14:29:36.284318 18526 ts_manager.cc:194] Registered new tserver with Master: d20e2fe413ea415a8bc10c379ae58d2e (127.0.98.1:46687)
I20260620 14:29:36.285997 18526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:29:36.292112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:36.395200 18722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:36.395450 18722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:36.395527 18722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:36.399555 18722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:36.399633 18722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:36.399721 18722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:29:36.404593 18722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:36.405597 18722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:36.406555 18722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:36.415377 18728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:36.415393 18730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:36.415539 18722 server_base.cc:1034] running on GCE node
W20260620 14:29:36.415386 18727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:36.416098 18722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:36.416675 18722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:36.417868 18722 hybrid_clock.cc:648] HybridClock initialized: now 1781965776417856 us; error 24 us; skew 500 ppm
Jun 20 14:29:36 dist-test-slave-pk3j krb5kdc[18481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:36.420317 18722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:29:36.421054 18722 webserver.cc:460] Webserver started at http://127.0.98.2:38165/ using document root <none> and password file <none>
I20260620 14:29:36.421588 18722 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:36.421648 18722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:36.421792 18722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:36.423589 18722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba52fd7536144bc19d9cef175943adde"
format_stamp: "Formatted at 2026-06-20 14:29:36 on dist-test-slave-pk3j"
server_key: "e6f9cf0ca95e0043cdf1246522fa1d03"
server_key_iv: "2776ea71db4345aed049a2b72362d7c8"
server_key_version: "encryptionkey@0"
I20260620 14:29:36.424075 18722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba52fd7536144bc19d9cef175943adde"
format_stamp: "Formatted at 2026-06-20 14:29:36 on dist-test-slave-pk3j"
server_key: "e6f9cf0ca95e0043cdf1246522fa1d03"
server_key_iv: "2776ea71db4345aed049a2b72362d7c8"
server_key_version: "encryptionkey@0"
I20260620 14:29:36.427525 18722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:29:36.429666 18737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:36.430783 18722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:29:36.430868 18722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "ba52fd7536144bc19d9cef175943adde"
format_stamp: "Formatted at 2026-06-20 14:29:36 on dist-test-slave-pk3j"
server_key: "e6f9cf0ca95e0043cdf1246522fa1d03"
server_key_iv: "2776ea71db4345aed049a2b72362d7c8"
server_key_version: "encryptionkey@0"
I20260620 14:29:36.430944 18722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:36.451284 18722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:36.453730 18722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:36.453861 18722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:36.454310 18722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:36.454999 18722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:36.455044 18722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:36.455080 18722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:36.455101 18722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:36.463255 18722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40005
I20260620 14:29:36.463295 18850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40005 every 8 connection(s)
I20260620 14:29:36.464051 18722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:29:36.467705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 18722
I20260620 14:29:36.467845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:29:36.468161 392 external_mini_cluster.cc:1428] Setting key ccd3e52683742a69e7db0e4f08d03729
Jun 20 14:29:36 dist-test-slave-pk3j krb5kdc[18481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:36.477041 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.465628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35655 (local address 127.0.98.62:33107)
0620 14:29:36.465770 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:36.465775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:36.466565 (+ 790us) server_negotiation.cc:407] Connection header received
0620 14:29:36.467309 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:36.467312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:36.467385 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:36.467472 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:36.468839 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.469475 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.470609 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.470854 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.470968 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.473290 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:36.473297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:36.473302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:36.473339 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:36.475042 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:36.475491 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:36.475495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:36.475497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:36.475592 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:36.475921 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:36.475926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:36.475927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:36.476168 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:36.476277 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:36.476714 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 14:29:36.476847 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 14:29:36.477728 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.466008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33107 (local address 127.0.98.2:35655)
0620 14:29:36.466409 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:36.466446 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:36.467127 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:36.467664 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:36.467674 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:36.468133 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:36.468703 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.468717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.469588 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.469592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.470484 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.470493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.470996 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.471000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.471163 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.471721 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:36.471736 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:36.473113 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:36.475151 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:36.475155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:36.475166 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:36.475393 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:36.475682 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:36.475684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:36.475685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:36.475818 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:36.476277 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:36.476282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:36.476494 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 14:29:36.476633 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 14:29:36.476863 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260620 14:29:36.478863 18851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33107
I20260620 14:29:36.479085 18851 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:36.479532 18851 heartbeater.cc:511] Master 127.0.98.62:33107 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 14:29:36.480515 18526 ts_manager.cc:194] Registered new tserver with Master: ba52fd7536144bc19d9cef175943adde (127.0.98.2:40005)
I20260620 14:29:36.481045 18526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:29:36.492820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:36.597249 18856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:36.597471 18856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:36.597510 18856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:36.601226 18856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:36.601290 18856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:36.601358 18856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:29:36.606480 18856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:36.607365 18856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:36.608211 18856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:36.616798 18862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:36.616969 18856 server_base.cc:1034] running on GCE node
W20260620 14:29:36.616811 18861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:36.617254 18864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:36.617643 18856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:36.618207 18856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:36.619391 18856 hybrid_clock.cc:648] HybridClock initialized: now 1781965776619383 us; error 28 us; skew 500 ppm
Jun 20 14:29:36 dist-test-slave-pk3j krb5kdc[18481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:36.626396 18856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:29:36.627177 18856 webserver.cc:460] Webserver started at http://127.0.98.3:40849/ using document root <none> and password file <none>
I20260620 14:29:36.627686 18856 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:36.627770 18856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:36.627985 18856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:36.629906 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c5ba25724a240b48a029202542548a9"
format_stamp: "Formatted at 2026-06-20 14:29:36 on dist-test-slave-pk3j"
server_key: "b98955c75a40f5899af6a4b5e5e2da74"
server_key_iv: "42c0ba78c5967fdd442a0e7694361024"
server_key_version: "encryptionkey@0"
I20260620 14:29:36.630429 18856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c5ba25724a240b48a029202542548a9"
format_stamp: "Formatted at 2026-06-20 14:29:36 on dist-test-slave-pk3j"
server_key: "b98955c75a40f5899af6a4b5e5e2da74"
server_key_iv: "42c0ba78c5967fdd442a0e7694361024"
server_key_version: "encryptionkey@0"
I20260620 14:29:36.633906 18856 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:29:36.636355 18871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:36.637491 18856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:29:36.637626 18856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "6c5ba25724a240b48a029202542548a9"
format_stamp: "Formatted at 2026-06-20 14:29:36 on dist-test-slave-pk3j"
server_key: "b98955c75a40f5899af6a4b5e5e2da74"
server_key_iv: "42c0ba78c5967fdd442a0e7694361024"
server_key_version: "encryptionkey@0"
I20260620 14:29:36.637743 18856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:36.655706 18856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:36.659080 18856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:36.659258 18856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:36.659785 18856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:36.660516 18856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:36.660580 18856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:36.660645 18856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:36.660693 18856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:36.668922 18856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38173
I20260620 14:29:36.668968 18984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38173 every 8 connection(s)
I20260620 14:29:36.669813 18856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:29:36.678297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 18856
Jun 20 14:29:36 dist-test-slave-pk3j krb5kdc[18481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:36.678434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:29:36.678718 392 external_mini_cluster.cc:1428] Setting key 93a37fed706adfa3b0dc8e9fcfc8f05e
I20260620 14:29:36.682210 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.671520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56263 (local address 127.0.98.62:33107)
0620 14:29:36.671653 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:36.671657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:36.672449 (+ 792us) server_negotiation.cc:407] Connection header received
0620 14:29:36.673718 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:36.673721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:36.673775 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:36.673850 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:36.674911 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.675425 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.676465 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.676696 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.676781 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.679043 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:36.679046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:36.679048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:36.679070 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:36.680501 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:36.680982 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:36.680984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:36.680986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:36.681058 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:36.681381 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:36.681383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:36.681385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:36.681538 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:36.681608 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:36.682013 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 14:29:36.682100 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:29:36.682890 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.671877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33107 (local address 127.0.98.3:56263)
0620 14:29:36.672280 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:36.672319 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:36.673521 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:36.674001 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:36.674007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:36.674337 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:36.674784 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.674797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.675523 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.675526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.676364 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.676373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.676823 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.676826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.676951 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.677444 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:36.677459 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:36.678881 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:36.680610 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:36.680616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:36.680629 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:36.680902 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:36.681133 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:36.681135 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:36.681137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:36.681313 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:36.681617 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:36.681622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:36.681817 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 14:29:36.681954 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:29:36.682169 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":106,"threads_started":1}
I20260620 14:29:36.683858 18985 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33107
I20260620 14:29:36.684078 18985 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:36.684513 18985 heartbeater.cc:511] Master 127.0.98.62:33107 requested a full tablet report, sending...
I20260620 14:29:36.685452 18526 ts_manager.cc:194] Registered new tserver with Master: 6c5ba25724a240b48a029202542548a9 (127.0.98.3:38173)
I20260620 14:29:36.685973 18526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56263
I20260620 14:29:36.692211 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:29:36.700872 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.694157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.0.98.62:33107)
0620 14:29:36.694304 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:36.694309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:36.694512 (+ 203us) server_negotiation.cc:407] Connection header received
0620 14:29:36.694646 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:36.694648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:36.694683 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:36.694753 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:36.695417 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.695889 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.696658 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.696863 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.696935 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.698343 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:36.698346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:36.698349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:36.698369 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:36.699477 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:36.699885 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:36.699888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:36.699889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:36.699952 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:36.700193 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:36.700195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:36.700196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:36.700356 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:36.700460 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:36.700623 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:29:36.700687 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:29:36.703377 18526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:29:36.706286 18526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:29:36.720242 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.714274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46687 (local address 127.0.0.1:58656)
0620 14:29:36.715033 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:36.715077 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:36.715204 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:36.716120 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:36.716126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:36.716151 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:36.716301 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.716308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.717411 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.717416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.718666 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.718677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.719759 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.719762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.719864 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.719887 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:29:36.719931 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:29:36.720065 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":33,"thread_start_us":79,"threads_started":1}
I20260620 14:29:36.720250 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.715453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.0.98.2:40005)
0620 14:29:36.715845 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:36.715854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:36.715872 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:29:36.715930 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:36.715934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:36.716100 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:36.716240 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:36.716628 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.717734 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.719049 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.719579 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.719702 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.719848 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:36.719987 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 14:29:36.720103 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":87,"threads_started":1}
I20260620 14:29:36.720621 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.714634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40005 (local address 127.0.0.1:52956)
0620 14:29:36.715406 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:36.715419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:36.715495 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:36.716398 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:36.716404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:36.716421 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:36.716489 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.716493 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.717902 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.717905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.718939 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.718945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.719700 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.719704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.719811 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.719843 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:29:36.719951 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:29:36.720091 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":45,"thread_start_us":81,"threads_started":1}
I20260620 14:29:36.720752 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.714350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38173 (local address 127.0.0.1:47820)
0620 14:29:36.715108 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:36.715124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:36.715205 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:36.716310 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:36.716312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:36.716327 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:36.716446 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.716452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.718313 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.718319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.719350 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.719361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.720411 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.720416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.720504 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.720524 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:29:36.720564 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 14:29:36.720624 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260620 14:29:36.721105 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.714420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.0.98.1:46687)
0620 14:29:36.715721 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:36.715732 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:36.715750 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:29:36.715812 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:36.715815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:36.715961 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:36.716113 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:36.716433 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.717286 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.718808 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.719629 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.720180 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.720652 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:36.720701 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:29:36.720806 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1204,"thread_start_us":125,"threads_started":1}
I20260620 14:29:36.721369 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.715029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.0.98.3:38173)
0620 14:29:36.715640 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:36.715652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:36.715676 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:29:36.715764 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:36.715769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:36.715961 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:36.716113 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:36.716570 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.718158 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.719502 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.720255 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.720926 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.721054 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:36.721104 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:29:36.721194 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":153,"threads_started":1}
I20260620 14:29:36.723555 18919 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2a684fe7fb4624a112c11c4c48f13a (DEFAULT_TABLE table=test-table [id=75202e19e48343e39040de2a0c3576de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:36.723598 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2a684fe7fb4624a112c11c4c48f13a (DEFAULT_TABLE table=test-table [id=75202e19e48343e39040de2a0c3576de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:36.723603 18785 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2a684fe7fb4624a112c11c4c48f13a (DEFAULT_TABLE table=test-table [id=75202e19e48343e39040de2a0c3576de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:36.724509 18919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2a684fe7fb4624a112c11c4c48f13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:36.724575 18785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2a684fe7fb4624a112c11c4c48f13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:36.724575 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2a684fe7fb4624a112c11c4c48f13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:36.730600 19002 tablet_bootstrap.cc:492] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: Bootstrap starting.
I20260620 14:29:36.731611 19003 tablet_bootstrap.cc:492] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: Bootstrap starting.
I20260620 14:29:36.732159 19004 tablet_bootstrap.cc:492] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e: Bootstrap starting.
I20260620 14:29:36.732933 19002 tablet_bootstrap.cc:654] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:36.733990 19004 tablet_bootstrap.cc:654] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:36.734118 19003 tablet_bootstrap.cc:654] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:36.734298 19002 log.cc:824] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:36.734948 19003 log.cc:824] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:36.735247 19004 log.cc:824] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:36.736377 19002 tablet_bootstrap.cc:492] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: No bootstrap required, opened a new log
I20260620 14:29:36.736560 19002 ts_tablet_manager.cc:1397] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:29:36.736927 19003 tablet_bootstrap.cc:492] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: No bootstrap required, opened a new log
I20260620 14:29:36.736936 19004 tablet_bootstrap.cc:492] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e: No bootstrap required, opened a new log
I20260620 14:29:36.737097 19003 ts_tablet_manager.cc:1397] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:29:36.737123 19004 ts_tablet_manager.cc:1397] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:29:36.739688 19004 raft_consensus.cc:348] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46687 } } peers { permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 } } peers { permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 } }
I20260620 14:29:36.739914 19004 raft_consensus.cc:374] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:36.739980 19004 raft_consensus.cc:729] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20e2fe413ea415a8bc10c379ae58d2e, State: Initialized, Role: FOLLOWER
I20260620 14:29:36.740355 19004 consensus_queue.cc:260] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46687 } } peers { permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 } } peers { permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 } }
I20260620 14:29:36.740387 19002 raft_consensus.cc:348] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46687 } } peers { permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 } } peers { permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 } }
I20260620 14:29:36.740635 19002 raft_consensus.cc:374] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:36.740695 19002 raft_consensus.cc:729] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5ba25724a240b48a029202542548a9, State: Initialized, Role: FOLLOWER
I20260620 14:29:36.740669 19003 raft_consensus.cc:348] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46687 } } peers { permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 } } peers { permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 } }
I20260620 14:29:36.740942 19003 raft_consensus.cc:374] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:36.741026 19003 raft_consensus.cc:729] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba52fd7536144bc19d9cef175943adde, State: Initialized, Role: FOLLOWER
I20260620 14:29:36.741124 19002 consensus_queue.cc:260] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46687 } } peers { permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 } } peers { permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 } }
I20260620 14:29:36.741310 18717 heartbeater.cc:503] Master 127.0.98.62:33107 was elected leader, sending a full tablet report...
I20260620 14:29:36.741520 19004 ts_tablet_manager.cc:1428] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 14:29:36.741694 19003 consensus_queue.cc:260] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46687 } } peers { permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 } } peers { permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 } }
I20260620 14:29:36.742059 19002 ts_tablet_manager.cc:1428] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260620 14:29:36.742547 19003 ts_tablet_manager.cc:1428] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:29:36.742638 18985 heartbeater.cc:503] Master 127.0.98.62:33107 was elected leader, sending a full tablet report...
I20260620 14:29:36.742820 18851 heartbeater.cc:503] Master 127.0.98.62:33107 was elected leader, sending a full tablet report...
W20260620 14:29:36.765935 18718 tablet.cc:2369] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:29:36.920794 18986 tablet.cc:2369] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:29:36.965078 18852 tablet.cc:2369] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:29:36.988510 19008 raft_consensus.cc:482] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:36.988690 19008 raft_consensus.cc:504] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46687 } } peers { permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 } } peers { permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 } }
I20260620 14:29:36.989831 19008 leader_election.cc:288] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c5ba25724a240b48a029202542548a9 (127.0.98.3:38173), ba52fd7536144bc19d9cef175943adde (127.0.98.2:40005)
I20260620 14:29:36.993991 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.990078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40005 (local address 127.0.98.1:53445)
0620 14:29:36.990421 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:36.990437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:36.990548 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:36.990831 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:36.990835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:36.990876 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:36.990998 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.991003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.991719 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.991726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.992760 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.992769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.993471 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.993476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.993578 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.993617 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:29:36.993703 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:29:36.993779 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":56,"spinlock_wait_cycles":768,"thread_start_us":117,"threads_started":1}
I20260620 14:29:36.994000 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.990194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53445 (local address 127.0.98.2:40005)
0620 14:29:36.990314 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:36.990318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:36.990531 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:29:36.990647 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:36.990649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:36.990696 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:36.990806 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:36.991142 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.991610 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.992875 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.993332 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.993471 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.993523 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:36.993746 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:29:36.993822 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:29:36.994174 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.990012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38173 (local address 127.0.98.1:54509)
0620 14:29:36.990326 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:36.990341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:36.990452 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:36.990783 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:36.990788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:36.990808 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:36.990975 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.990981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.991721 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.991725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.992768 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:36.992777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.993647 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.993649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:36.993811 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.993833 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:29:36.993951 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:29:36.994024 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":40,"spinlock_wait_cycles":17024,"thread_start_us":68,"threads_started":1}
I20260620 14:29:36.994195 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:36.990120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54509 (local address 127.0.98.3:38173)
0620 14:29:36.990256 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:36.990261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:36.990439 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:29:36.990587 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:36.990592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:36.990659 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:36.990761 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:36.991096 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.991608 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.992919 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:36.993535 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:36.993645 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:36.993735 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:36.993975 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:29:36.994078 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:29:36.994828 18939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a684fe7fb4624a112c11c4c48f13a" candidate_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5ba25724a240b48a029202542548a9" is_pre_election: true
I20260620 14:29:36.995086 18805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a684fe7fb4624a112c11c4c48f13a" candidate_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba52fd7536144bc19d9cef175943adde" is_pre_election: true
I20260620 14:29:36.995229 18939 raft_consensus.cc:2454] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20e2fe413ea415a8bc10c379ae58d2e in term 0.
I20260620 14:29:36.995424 18805 raft_consensus.cc:2454] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d20e2fe413ea415a8bc10c379ae58d2e in term 0.
I20260620 14:29:36.995756 18606 leader_election.cc:302] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5ba25724a240b48a029202542548a9, d20e2fe413ea415a8bc10c379ae58d2e; no voters:
I20260620 14:29:36.996065 19008 raft_consensus.cc:2781] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:29:36.996147 19008 raft_consensus.cc:482] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:29:36.996189 19008 raft_consensus.cc:3037] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:36.997176 19008 raft_consensus.cc:504] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46687 } } peers { permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 } } peers { permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 } }
I20260620 14:29:36.997529 19008 leader_election.cc:288] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [CANDIDATE]: Term 1 election: Requested vote from peers 6c5ba25724a240b48a029202542548a9 (127.0.98.3:38173), ba52fd7536144bc19d9cef175943adde (127.0.98.2:40005)
I20260620 14:29:36.997853 18939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a684fe7fb4624a112c11c4c48f13a" candidate_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c5ba25724a240b48a029202542548a9"
I20260620 14:29:36.997895 18805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2a684fe7fb4624a112c11c4c48f13a" candidate_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba52fd7536144bc19d9cef175943adde"
I20260620 14:29:36.997967 18939 raft_consensus.cc:3037] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:36.998025 18805 raft_consensus.cc:3037] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:36.998975 18939 raft_consensus.cc:2454] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20e2fe413ea415a8bc10c379ae58d2e in term 1.
I20260620 14:29:36.999203 18805 raft_consensus.cc:2454] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d20e2fe413ea415a8bc10c379ae58d2e in term 1.
I20260620 14:29:36.999215 18606 leader_election.cc:302] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c5ba25724a240b48a029202542548a9, d20e2fe413ea415a8bc10c379ae58d2e; no voters:
I20260620 14:29:36.999366 19008 raft_consensus.cc:2781] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:36.999598 19008 raft_consensus.cc:686] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [term 1 LEADER]: Becoming Leader. State: Replica: d20e2fe413ea415a8bc10c379ae58d2e, State: Running, Role: LEADER
I20260620 14:29:36.999931 19008 consensus_queue.cc:237] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46687 } } peers { permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 } } peers { permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 } }
I20260620 14:29:37.003058 18524 catalog_manager.cc:5495] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e reported cstate change: term changed from 0 to 1, leader changed from <none> to d20e2fe413ea415a8bc10c379ae58d2e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d20e2fe413ea415a8bc10c379ae58d2e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 } health_report { overall_health: UNKNOWN } } }
I20260620 14:29:37.080853 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.077344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.0.98.1:46687)
0620 14:29:37.077490 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:37.077495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:37.077522 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:29:37.077608 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:37.077610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:37.077662 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:37.077778 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:29:37.078037 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.078562 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.079397 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.079612 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.079719 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:37.080044 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:29:37.080445 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:29:37.080611 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:37.080674 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:29:37.080707 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:29:37.086553 18939 raft_consensus.cc:1261] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9 [term 1 FOLLOWER]: Refusing update from remote peer d20e2fe413ea415a8bc10c379ae58d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:37.086561 18805 raft_consensus.cc:1261] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde [term 1 FOLLOWER]: Refusing update from remote peer d20e2fe413ea415a8bc10c379ae58d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:37.087260 19013 consensus_queue.cc:1035] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c5ba25724a240b48a029202542548a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:37.087435 19008 consensus_queue.cc:1035] T 9b2a684fe7fb4624a112c11c4c48f13a P d20e2fe413ea415a8bc10c379ae58d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba52fd7536144bc19d9cef175943adde" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:37.093959 19016 mvcc.cc:205] Tried to move back new op lower bound from 7298931822938800128 to 7298931822594260992. Current Snapshot: MvccSnapshot[applied={T|T < 7298931822938800128}]
I20260620 14:29:37.095147 19017 mvcc.cc:205] Tried to move back new op lower bound from 7298931822938800128 to 7298931822594260992. Current Snapshot: MvccSnapshot[applied={T|T < 7298931822938800128}]
I20260620 14:29:37.097178 19018 mvcc.cc:205] Tried to move back new op lower bound from 7298931822938800128 to 7298931822594260992. Current Snapshot: MvccSnapshot[applied={T|T < 7298931822938800128 or (T in {7298931822938800128})}]
I20260620 14:29:37.103050 18526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:37.103185 18526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:37.105399 18526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a3379517c384d2d8f8d7e4a5d3f4aeb: Sending DeleteTablet for 3 replicas of tablet 9b2a684fe7fb4624a112c11c4c48f13a
I20260620 14:29:37.106102 18919 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2a684fe7fb4624a112c11c4c48f13a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47820
I20260620 14:29:37.106281 18651 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2a684fe7fb4624a112c11c4c48f13a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58656
I20260620 14:29:37.106643 19025 tablet_replica.cc:330] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: stopping tablet replica
I20260620 14:29:37.106863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 18588
I20260620 14:29:37.106820 18785 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2a684fe7fb4624a112c11c4c48f13a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52956
I20260620 14:29:37.106935 19025 raft_consensus.cc:2229] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:37.107203 19025 raft_consensus.cc:2258] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:37.108629 19025 ts_tablet_manager.cc:1905] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:37.108752 19027 tablet_replica.cc:330] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: stopping tablet replica
I20260620 14:29:37.109026 19027 raft_consensus.cc:2229] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:37.109342 19027 raft_consensus.cc:2258] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:37.110532 19027 ts_tablet_manager.cc:1905] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:37.112064 19025 ts_tablet_manager.cc:1918] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:37.112164 19025 log.cc:1196] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/9b2a684fe7fb4624a112c11c4c48f13a
I20260620 14:29:37.112499 19025 ts_tablet_manager.cc:1939] T 9b2a684fe7fb4624a112c11c4c48f13a P 6c5ba25724a240b48a029202542548a9: Deleting consensus metadata
I20260620 14:29:37.112753 19027 ts_tablet_manager.cc:1918] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:37.112865 19027 log.cc:1196] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/9b2a684fe7fb4624a112c11c4c48f13a
I20260620 14:29:37.113147 19027 ts_tablet_manager.cc:1939] T 9b2a684fe7fb4624a112c11c4c48f13a P ba52fd7536144bc19d9cef175943adde: Deleting consensus metadata
I20260620 14:29:37.113745 18512 catalog_manager.cc:4841] TS 6c5ba25724a240b48a029202542548a9 (127.0.98.3:38173): tablet 9b2a684fe7fb4624a112c11c4c48f13a (table test-table [id=75202e19e48343e39040de2a0c3576de]) successfully deleted
I20260620 14:29:37.114140 18513 catalog_manager.cc:4841] TS ba52fd7536144bc19d9cef175943adde (127.0.98.2:40005): tablet 9b2a684fe7fb4624a112c11c4c48f13a (table test-table [id=75202e19e48343e39040de2a0c3576de]) successfully deleted
W20260620 14:29:37.114878 18510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46687 (error 108)
I20260620 14:29:37.115068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 18722
I20260620 14:29:37.116230 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.115977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46687 (local address 127.0.0.1:58676)
0620 14:29:37.116100 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:37.116164 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:29:37.116369 18510 catalog_manager.cc:4584] TS d20e2fe413ea415a8bc10c379ae58d2e (127.0.98.1:46687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b2a684fe7fb4624a112c11c4c48f13a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46687: connect: Connection refused (error 111)
I20260620 14:29:37.121459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 18856
I20260620 14:29:37.126801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 18494
2026-06-20T14:29:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2461 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:29:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:37Z Disabled control of system clock
I20260620 14:29:37.155767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42501
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:29:37.262639 19032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:37.262880 19032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:37.262953 19032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:37.266873 19032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:37.266932 19032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:37.266948 19032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:37.267005 19032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:37.267027 19032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:37.271721 19032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:37.272702 19032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:37.274060 19032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:37.281904 19040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:37.281915 19038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:37.282071 19032 server_base.cc:1034] running on GCE node
W20260620 14:29:37.281913 19037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:37.282809 19032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:37.283803 19032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:37.284988 19032 hybrid_clock.cc:648] HybridClock initialized: now 1781965777284961 us; error 43 us; skew 500 ppm
I20260620 14:29:37.286734 19032 webserver.cc:460] Webserver started at http://127.0.98.62:39831/ using document root <none> and password file <none>
I20260620 14:29:37.287286 19032 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:37.287374 19032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:37.287569 19032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:37.289534 19032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "b843455ea9c948b9bd884a6d495dc77b"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "c787fe4df7cafeb8a39bfd60172f218f"
server_key_iv: "d4e52ccd1bf5ba1dcd721253bef6fdaa"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.290077 19032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "b843455ea9c948b9bd884a6d495dc77b"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "c787fe4df7cafeb8a39bfd60172f218f"
server_key_iv: "d4e52ccd1bf5ba1dcd721253bef6fdaa"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.293566 19032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:29:37.295800 19046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:37.296911 19032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:37.297043 19032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "b843455ea9c948b9bd884a6d495dc77b"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "c787fe4df7cafeb8a39bfd60172f218f"
server_key_iv: "d4e52ccd1bf5ba1dcd721253bef6fdaa"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.297148 19032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:37.326822 19032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:37.327529 19032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:37.327729 19032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:37.334599 19032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42501
I20260620 14:29:37.334607 19098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42501 every 8 connection(s)
I20260620 14:29:37.335480 19032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:37.338380 19099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:37.341486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 19032
I20260620 14:29:37.341622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:37.341907 392 external_mini_cluster.cc:1428] Setting key edadd467dde0d49289b1d74a3d050ba5
I20260620 14:29:37.344074 19099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b: Bootstrap starting.
I20260620 14:29:37.346189 19099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:37.346873 19099 log.cc:824] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:37.348592 19099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b: No bootstrap required, opened a new log
I20260620 14:29:37.350659 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.343246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.0.98.62:42501)
0620 14:29:37.343883 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:37.343905 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:37.343940 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:29:37.344583 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:37.344600 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:37.344990 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:37.345328 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:37.345727 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.346687 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.347569 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.348031 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.348195 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:37.349034 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:37.349039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:37.349049 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:37.349484 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:37.349534 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:37.349555 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:37.349662 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:37.349800 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:29:37.350033 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":147,"threads_started":1}
I20260620 14:29:37.352161 19099 raft_consensus.cc:348] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b843455ea9c948b9bd884a6d495dc77b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } }
I20260620 14:29:37.352350 19099 raft_consensus.cc:374] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:37.352416 19099 raft_consensus.cc:729] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b843455ea9c948b9bd884a6d495dc77b, State: Initialized, Role: FOLLOWER
I20260620 14:29:37.352852 19099 consensus_queue.cc:260] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b843455ea9c948b9bd884a6d495dc77b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } }
I20260620 14:29:37.352988 19099 raft_consensus.cc:388] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:37.353065 19099 raft_consensus.cc:482] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:37.353185 19099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:37.354014 19099 raft_consensus.cc:504] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b843455ea9c948b9bd884a6d495dc77b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } }
I20260620 14:29:37.354329 19099 leader_election.cc:302] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b843455ea9c948b9bd884a6d495dc77b; no voters:
I20260620 14:29:37.354596 19099 leader_election.cc:288] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:37.354704 19104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:37.355198 19104 raft_consensus.cc:686] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [term 1 LEADER]: Becoming Leader. State: Replica: b843455ea9c948b9bd884a6d495dc77b, State: Running, Role: LEADER
I20260620 14:29:37.355504 19104 consensus_queue.cc:237] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b843455ea9c948b9bd884a6d495dc77b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } }
I20260620 14:29:37.355854 19099 sys_catalog.cc:564] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:37.356978 19106 sys_catalog.cc:455] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b843455ea9c948b9bd884a6d495dc77b. Latest consensus state: current_term: 1 leader_uuid: "b843455ea9c948b9bd884a6d495dc77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b843455ea9c948b9bd884a6d495dc77b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } } }
I20260620 14:29:37.357097 19106 sys_catalog.cc:458] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:37.357393 19105 sys_catalog.cc:455] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b843455ea9c948b9bd884a6d495dc77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b843455ea9c948b9bd884a6d495dc77b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42501 } } }
I20260620 14:29:37.357501 19105 sys_catalog.cc:458] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:37.357542 19113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:37.360474 19113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:37.365604 19113 catalog_manager.cc:1347] Generated new cluster ID: 5b3fe977606149fbaf84f99e53fa3e18
I20260620 14:29:37.365664 19113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:37.393438 19113 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:37.394655 19113 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:37.399497 19113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b: Generated new TSK 0
I20260620 14:29:37.400120 19113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:37.409457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--builtin_ntp_servers=127.0.98.20:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:29:37.530025 19123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:37.530262 19123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:37.530298 19123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:37.534180 19123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:37.534231 19123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:37.534282 19123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:29:37.538796 19123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:37.539659 19123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:37.540870 19123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:37.549552 19129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:37.549587 19128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:37.549724 19123 server_base.cc:1034] running on GCE node
W20260620 14:29:37.549563 19131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:37.550185 19123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:37.550832 19123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:37.552016 19123 hybrid_clock.cc:648] HybridClock initialized: now 1781965777552002 us; error 29 us; skew 500 ppm
I20260620 14:29:37.553759 19123 webserver.cc:460] Webserver started at http://127.0.98.1:35307/ using document root <none> and password file <none>
I20260620 14:29:37.554316 19123 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:37.554409 19123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:37.554625 19123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:37.556929 19123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "63362f25483c4b0a9cf78331da1f9674"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "9cb5e5923faa1e796f85031eea2000be"
server_key_iv: "6711be92dad3815280554d9c9f93e285"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.557561 19123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "63362f25483c4b0a9cf78331da1f9674"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "9cb5e5923faa1e796f85031eea2000be"
server_key_iv: "6711be92dad3815280554d9c9f93e285"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.561190 19123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:29:37.563570 19137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:37.564738 19123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:29:37.564867 19123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "63362f25483c4b0a9cf78331da1f9674"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "9cb5e5923faa1e796f85031eea2000be"
server_key_iv: "6711be92dad3815280554d9c9f93e285"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.564975 19123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:37.578071 19123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:37.578701 19123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:37.578887 19123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:37.579497 19123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:37.580391 19123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:37.580461 19123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:37.580543 19123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:37.580585 19123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:37.588977 19123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42255
I20260620 14:29:37.588999 19250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42255 every 8 connection(s)
I20260620 14:29:37.589864 19123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:29:37.596272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 19123
I20260620 14:29:37.596407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:29:37.596701 392 external_mini_cluster.cc:1428] Setting key b69fcfb81580345345af2934c00a2a94
I20260620 14:29:37.598810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--builtin_ntp_servers=127.0.98.20:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:29:37.599687 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.591636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37963 (local address 127.0.98.62:42501)
0620 14:29:37.591807 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:37.591811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:37.592878 (+ 1067us) server_negotiation.cc:407] Connection header received
0620 14:29:37.594334 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:37.594337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:37.594382 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:37.594456 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:37.595353 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.595896 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.597578 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.597834 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.597947 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:37.599160 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:37.599163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:37.599165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:37.599177 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:37.599195 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:37.599203 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:37.599263 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:37.599455 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:29:37.599529 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:29:37.600713 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.592105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42501 (local address 127.0.98.1:37963)
0620 14:29:37.592727 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:37.592767 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:37.594131 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:37.594613 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:37.594620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:37.594705 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:37.595172 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:37.595209 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.596084 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.596089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:37.597454 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:37.597463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.597993 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.597997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:37.598196 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:37.598809 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:29:37.598812 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:37.598866 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:29:37.598952 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:37.599298 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:37.599309 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:37.599318 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:29:37.599447 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:29:37.599756 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260620 14:29:37.601967 19251 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42501
I20260620 14:29:37.602200 19251 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:37.602694 19251 heartbeater.cc:511] Master 127.0.98.62:42501 requested a full tablet report, sending...
I20260620 14:29:37.604323 19063 ts_manager.cc:194] Registered new tserver with Master: 63362f25483c4b0a9cf78331da1f9674 (127.0.98.1:42255)
I20260620 14:29:37.606076 19063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37963
W20260620 14:29:37.706720 19254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:37.706945 19254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:37.706998 19254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:37.710951 19254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:37.710997 19254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:37.711047 19254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:29:37.715531 19254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:37.716452 19254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:37.717697 19254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:37.726634 19262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:37.726658 19260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:37.726658 19259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:37.726956 19254 server_base.cc:1034] running on GCE node
I20260620 14:29:37.727368 19254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:37.727972 19254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:37.729153 19254 hybrid_clock.cc:648] HybridClock initialized: now 1781965777729144 us; error 29 us; skew 500 ppm
I20260620 14:29:37.730949 19254 webserver.cc:460] Webserver started at http://127.0.98.2:42535/ using document root <none> and password file <none>
I20260620 14:29:37.731493 19254 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:37.731581 19254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:37.731807 19254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:37.733812 19254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce2b93a688c546cc923ea8df822ca21e"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "a05990d2ed5961ef9eece2ea55a2e0cc"
server_key_iv: "c52762a4d9911fadfdadad91ab9a554b"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.734364 19254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce2b93a688c546cc923ea8df822ca21e"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "a05990d2ed5961ef9eece2ea55a2e0cc"
server_key_iv: "c52762a4d9911fadfdadad91ab9a554b"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.737733 19254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:29:37.740065 19268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:37.741194 19254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:37.741369 19254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "ce2b93a688c546cc923ea8df822ca21e"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "a05990d2ed5961ef9eece2ea55a2e0cc"
server_key_iv: "c52762a4d9911fadfdadad91ab9a554b"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.741472 19254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:37.768602 19254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:37.769400 19254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:37.769618 19254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:37.770202 19254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:37.771154 19254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:37.771227 19254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:37.771293 19254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:37.771351 19254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:37.780149 19254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39205
I20260620 14:29:37.780179 19381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39205 every 8 connection(s)
I20260620 14:29:37.781005 19254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:29:37.785322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 19254
I20260620 14:29:37.785439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:29:37.785742 392 external_mini_cluster.cc:1428] Setting key 8a73baf8c7734bc5b4c6c8c07f88cae6
I20260620 14:29:37.787648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--builtin_ntp_servers=127.0.98.20:39919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:29:37.790499 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.782760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41887 (local address 127.0.98.62:42501)
0620 14:29:37.782910 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:37.782915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:37.783982 (+ 1067us) server_negotiation.cc:407] Connection header received
0620 14:29:37.785315 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:37.785318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:37.785377 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:37.785478 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:37.786482 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.787076 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.788305 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.788690 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.788810 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:37.790076 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:37.790079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:37.790081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:37.790093 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:37.790110 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:37.790117 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:37.790177 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:37.790343 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:29:37.790393 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:29:37.791728 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.783193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42501 (local address 127.0.98.2:41887)
0620 14:29:37.783811 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:37.783849 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:37.785103 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:37.785635 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:37.785642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:37.785723 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:37.786341 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:37.786365 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.787218 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.787221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:37.788176 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:37.788186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.788871 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.788876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:37.789102 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:37.789757 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:29:37.789762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:37.789819 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:29:37.789919 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:37.790192 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:37.790203 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:37.790212 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:29:37.790340 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:29:37.790636 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20260620 14:29:37.792868 19382 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42501
I20260620 14:29:37.793100 19382 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:37.793689 19382 heartbeater.cc:511] Master 127.0.98.62:42501 requested a full tablet report, sending...
I20260620 14:29:37.794786 19063 ts_manager.cc:194] Registered new tserver with Master: ce2b93a688c546cc923ea8df822ca21e (127.0.98.2:39205)
I20260620 14:29:37.795658 19063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41887
W20260620 14:29:37.896099 19385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:37.896370 19385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:37.896407 19385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:37.900534 19385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:37.900583 19385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:37.900636 19385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:29:37.905315 19385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:37.906266 19385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:37.907585 19385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:37.916818 19390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:37.916808 19391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:37.916908 19385 server_base.cc:1034] running on GCE node
W20260620 14:29:37.916807 19393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:37.917480 19385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:37.918022 19385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:37.919196 19385 hybrid_clock.cc:648] HybridClock initialized: now 1781965777919170 us; error 47 us; skew 500 ppm
I20260620 14:29:37.921105 19385 webserver.cc:460] Webserver started at http://127.0.98.3:41193/ using document root <none> and password file <none>
I20260620 14:29:37.921877 19385 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:37.921964 19385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:37.922169 19385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:37.924170 19385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "19df7f5cfabb42caa62828004a614a3a"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "6817f8be97d421b2ccd2733217a78970"
server_key_iv: "9caeae836e3ba2c49f326b11a1bdafba"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.924729 19385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "19df7f5cfabb42caa62828004a614a3a"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "6817f8be97d421b2ccd2733217a78970"
server_key_iv: "9caeae836e3ba2c49f326b11a1bdafba"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.928342 19385 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 14:29:37.930641 19399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:37.931874 19385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:29:37.932006 19385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "19df7f5cfabb42caa62828004a614a3a"
format_stamp: "Formatted at 2026-06-20 14:29:37 on dist-test-slave-pk3j"
server_key: "6817f8be97d421b2ccd2733217a78970"
server_key_iv: "9caeae836e3ba2c49f326b11a1bdafba"
server_key_version: "encryptionkey@0"
I20260620 14:29:37.932132 19385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:37.948390 19385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:37.949041 19385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:37.949294 19385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:37.949908 19385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:37.950834 19385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:37.950904 19385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:37.950968 19385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:37.951018 19385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:37.960842 19385 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44473
I20260620 14:29:37.960862 19512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44473 every 8 connection(s)
I20260620 14:29:37.961732 19385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:29:37.964241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 19385
I20260620 14:29:37.964363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:29:37.964663 392 external_mini_cluster.cc:1428] Setting key 423dd294bdfe0b98e6f859183d8da35a
I20260620 14:29:37.970624 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.963455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51105 (local address 127.0.98.62:42501)
0620 14:29:37.963579 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:37.963582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:37.964746 (+ 1164us) server_negotiation.cc:407] Connection header received
0620 14:29:37.966231 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:37.966234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:37.966284 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:37.966364 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:37.967232 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.967726 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.968863 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.969117 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.969196 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:37.970240 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:37.970243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:37.970245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:37.970255 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:37.970272 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:37.970279 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:37.970335 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:37.970471 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 14:29:37.970519 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:29:37.971438 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.963833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42501 (local address 127.0.98.3:51105)
0620 14:29:37.964577 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:37.964630 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:37.966052 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:37.966489 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:37.966495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:37.966573 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:37.967114 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:37.967127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.967832 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.967834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:37.968757 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:37.968766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.969277 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.969281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:37.969414 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:37.970013 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:29:37.970015 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:37.970051 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:29:37.970108 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:37.970351 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:37.970356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:37.970360 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:29:37.970450 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:29:37.970651 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20260620 14:29:37.972414 19513 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42501
I20260620 14:29:37.972602 19513 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:37.973024 19513 heartbeater.cc:511] Master 127.0.98.62:42501 requested a full tablet report, sending...
I20260620 14:29:37.973953 19063 ts_manager.cc:194] Registered new tserver with Master: 19df7f5cfabb42caa62828004a614a3a (127.0.98.3:44473)
I20260620 14:29:37.974458 19063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51105
I20260620 14:29:37.978298 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:29:37.983850 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.980038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.0.98.62:42501)
0620 14:29:37.980179 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:37.980183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:37.980348 (+ 165us) server_negotiation.cc:407] Connection header received
0620 14:29:37.980494 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:37.980497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:37.980537 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:37.980621 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:37.980877 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.981430 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.982235 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:37.982459 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:37.982579 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:37.983340 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:37.983344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:37.983346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:37.983358 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:37.983376 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:37.983391 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:37.983515 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:37.983655 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:29:37.983690 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:29:37.986459 19063 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:29:37.989166 19063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:29:38.003300 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.998094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.0.98.1:42255)
0620 14:29:37.998376 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:37.998397 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:37.998617 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:29:37.998922 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:37.998927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:37.999104 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:37.999248 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:37.999474 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.000570 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.002083 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.002671 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.002804 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.002994 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:38.003052 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:29:38.003141 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":111,"threads_started":1}
I20260620 14:29:38.003445 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.997271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44473 (local address 127.0.0.1:40234)
0620 14:29:37.997608 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:37.997658 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:37.997820 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:37.998820 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:37.998827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:37.998849 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:37.998996 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:37.999003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.000432 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.000435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:38.001963 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:38.001975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.003033 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.003036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:38.003123 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.003145 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:29:38.003195 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:29:38.003266 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":99,"thread_start_us":83,"threads_started":1}
I20260620 14:29:38.003695 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.997983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39205 (local address 127.0.0.1:44230)
0620 14:29:37.998910 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:37.998921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:37.999004 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:37.999417 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:37.999420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:37.999437 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:37.999529 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:37.999534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.000722 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.000725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:38.001993 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:38.002003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.003244 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.003247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:38.003343 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.003365 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:29:38.003428 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:29:38.003567 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":122,"thread_start_us":89,"threads_started":1}
I20260620 14:29:38.003892 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.997395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.0.98.3:44473)
0620 14:29:37.998317 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:37.998330 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:37.998355 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:29:37.998440 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:37.998445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:37.998666 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:37.998836 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:37.999127 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.000303 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.002105 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.002892 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.003398 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.003586 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:38.003634 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:29:38.003733 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":109,"threads_started":1}
I20260620 14:29:38.004130 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.998303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.0.98.2:39205)
0620 14:29:37.998615 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:37.998626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:37.998996 (+ 370us) server_negotiation.cc:407] Connection header received
0620 14:29:37.999099 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:37.999103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:37.999259 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:37.999386 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:37.999639 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.000630 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.002114 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.002900 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.003009 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.003187 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:38.003857 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 14:29:38.003976 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":105,"threads_started":1}
I20260620 14:29:38.003311 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:37.997665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42255 (local address 127.0.0.1:52840)
0620 14:29:37.998520 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:37.998535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:37.998696 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:37.999237 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:37.999239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:37.999257 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:37.999341 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:37.999346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.000678 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.000684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:38.001963 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:38.001972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.002794 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.002797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:38.002896 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.002930 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:29:38.003029 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:29:38.003141 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":76,"thread_start_us":111,"threads_started":1}
I20260620 14:29:38.006978 19445 tablet_service.cc:1467] Processing CreateTablet for tablet e47752a898a744c687cb3e50099642b5 (DEFAULT_TABLE table=test-table [id=0abb0cd9847c479a9d865e6fa4f8b9eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:38.007228 19185 tablet_service.cc:1467] Processing CreateTablet for tablet e47752a898a744c687cb3e50099642b5 (DEFAULT_TABLE table=test-table [id=0abb0cd9847c479a9d865e6fa4f8b9eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:38.008082 19445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47752a898a744c687cb3e50099642b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:38.008082 19185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47752a898a744c687cb3e50099642b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:38.009023 19316 tablet_service.cc:1467] Processing CreateTablet for tablet e47752a898a744c687cb3e50099642b5 (DEFAULT_TABLE table=test-table [id=0abb0cd9847c479a9d865e6fa4f8b9eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:38.009900 19316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47752a898a744c687cb3e50099642b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:38.015259 19531 tablet_bootstrap.cc:492] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: Bootstrap starting.
I20260620 14:29:38.015668 19530 tablet_bootstrap.cc:492] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674: Bootstrap starting.
I20260620 14:29:38.016424 19532 tablet_bootstrap.cc:492] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: Bootstrap starting.
I20260620 14:29:38.018041 19530 tablet_bootstrap.cc:654] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:38.018548 19532 tablet_bootstrap.cc:654] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:38.018926 19530 log.cc:824] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:38.019125 19531 tablet_bootstrap.cc:654] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:38.019330 19532 log.cc:824] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:38.019932 19531 log.cc:824] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:38.021014 19530 tablet_bootstrap.cc:492] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674: No bootstrap required, opened a new log
I20260620 14:29:38.021162 19532 tablet_bootstrap.cc:492] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: No bootstrap required, opened a new log
I20260620 14:29:38.021237 19530 ts_tablet_manager.cc:1397] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:29:38.021353 19532 ts_tablet_manager.cc:1397] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 14:29:38.021659 19531 tablet_bootstrap.cc:492] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: No bootstrap required, opened a new log
I20260620 14:29:38.021858 19531 ts_tablet_manager.cc:1397] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:29:38.025157 19531 raft_consensus.cc:348] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19df7f5cfabb42caa62828004a614a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44473 } } peers { permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 } } peers { permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 } }
I20260620 14:29:38.025461 19531 raft_consensus.cc:374] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:38.025514 19531 raft_consensus.cc:729] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19df7f5cfabb42caa62828004a614a3a, State: Initialized, Role: FOLLOWER
I20260620 14:29:38.025539 19532 raft_consensus.cc:348] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19df7f5cfabb42caa62828004a614a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44473 } } peers { permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 } } peers { permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 } }
I20260620 14:29:38.025539 19530 raft_consensus.cc:348] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19df7f5cfabb42caa62828004a614a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44473 } } peers { permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 } } peers { permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 } }
I20260620 14:29:38.025770 19532 raft_consensus.cc:374] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:38.025770 19530 raft_consensus.cc:374] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:38.025830 19532 raft_consensus.cc:729] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2b93a688c546cc923ea8df822ca21e, State: Initialized, Role: FOLLOWER
I20260620 14:29:38.025837 19530 raft_consensus.cc:729] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63362f25483c4b0a9cf78331da1f9674, State: Initialized, Role: FOLLOWER
I20260620 14:29:38.025991 19531 consensus_queue.cc:260] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19df7f5cfabb42caa62828004a614a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44473 } } peers { permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 } } peers { permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 } }
I20260620 14:29:38.026232 19530 consensus_queue.cc:260] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19df7f5cfabb42caa62828004a614a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44473 } } peers { permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 } } peers { permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 } }
I20260620 14:29:38.026232 19532 consensus_queue.cc:260] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19df7f5cfabb42caa62828004a614a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44473 } } peers { permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 } } peers { permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 } }
I20260620 14:29:38.026690 19513 heartbeater.cc:503] Master 127.0.98.62:42501 was elected leader, sending a full tablet report...
I20260620 14:29:38.026990 19531 ts_tablet_manager.cc:1428] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:29:38.027122 19530 ts_tablet_manager.cc:1428] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:29:38.027122 19532 ts_tablet_manager.cc:1428] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260620 14:29:38.027443 19251 heartbeater.cc:503] Master 127.0.98.62:42501 was elected leader, sending a full tablet report...
I20260620 14:29:38.028270 19382 heartbeater.cc:503] Master 127.0.98.62:42501 was elected leader, sending a full tablet report...
W20260620 14:29:38.031901 19383 tablet.cc:2369] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:29:38.091043 19252 tablet.cc:2369] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:29:38.212741 19514 tablet.cc:2369] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:29:38.214663 19536 raft_consensus.cc:482] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:38.214915 19536 raft_consensus.cc:504] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19df7f5cfabb42caa62828004a614a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44473 } } peers { permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 } } peers { permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 } }
I20260620 14:29:38.216436 19536 leader_election.cc:288] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce2b93a688c546cc923ea8df822ca21e (127.0.98.2:39205), 63362f25483c4b0a9cf78331da1f9674 (127.0.98.1:42255)
I20260620 14:29:38.220748 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:38.216724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42255 (local address 127.0.98.3:58165)
0620 14:29:38.216846 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:38.216864 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:38.216969 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:38.217439 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:38.217442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:38.217454 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:38.217592 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:38.217605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.218631 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.218637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:38.219682 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:38.219693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.220334 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.220338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:38.220434 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.220456 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:29:38.220551 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:29:38.220618 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:29:38.220748 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:38.217063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58165 (local address 127.0.98.1:42255)
0620 14:29:38.217185 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:38.217191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:38.217231 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:29:38.217301 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:38.217304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:38.217361 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:38.217434 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:38.217762 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.218552 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.219817 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.220212 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.220332 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.220390 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:38.220565 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:29:38.220635 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:29:38.221482 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:38.217803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35795 (local address 127.0.98.2:39205)
0620 14:29:38.217992 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:38.217999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:38.218017 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:29:38.218076 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:38.218082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:38.218145 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:38.218235 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:38.218556 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.219328 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.220306 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.220701 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.220820 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.220909 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:38.221333 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 14:29:38.221393 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:29:38.221448 19205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47752a898a744c687cb3e50099642b5" candidate_uuid: "19df7f5cfabb42caa62828004a614a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63362f25483c4b0a9cf78331da1f9674" is_pre_election: true
I20260620 14:29:38.221690 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:38.216744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39205 (local address 127.0.98.3:35795)
0620 14:29:38.217186 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:38.217220 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:38.217311 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:38.218247 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:38.218250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:38.218270 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:38.218396 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:38.218402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.219440 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.219446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:38.220182 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:38.220191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.221079 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.221082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:38.221192 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.221234 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:29:38.221315 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:29:38.221379 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":36,"thread_start_us":203,"threads_started":1}
I20260620 14:29:38.221755 19205 raft_consensus.cc:2454] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19df7f5cfabb42caa62828004a614a3a in term 0.
I20260620 14:29:38.222252 19401 leader_election.cc:302] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19df7f5cfabb42caa62828004a614a3a, 63362f25483c4b0a9cf78331da1f9674; no voters:
I20260620 14:29:38.222249 19336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47752a898a744c687cb3e50099642b5" candidate_uuid: "19df7f5cfabb42caa62828004a614a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2b93a688c546cc923ea8df822ca21e" is_pre_election: true
I20260620 14:29:38.222522 19536 raft_consensus.cc:2781] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:29:38.222522 19336 raft_consensus.cc:2454] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19df7f5cfabb42caa62828004a614a3a in term 0.
I20260620 14:29:38.222613 19536 raft_consensus.cc:482] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:29:38.222671 19536 raft_consensus.cc:3037] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:38.223738 19536 raft_consensus.cc:504] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19df7f5cfabb42caa62828004a614a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44473 } } peers { permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 } } peers { permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 } }
I20260620 14:29:38.224104 19536 leader_election.cc:288] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [CANDIDATE]: Term 1 election: Requested vote from peers ce2b93a688c546cc923ea8df822ca21e (127.0.98.2:39205), 63362f25483c4b0a9cf78331da1f9674 (127.0.98.1:42255)
I20260620 14:29:38.224423 19336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47752a898a744c687cb3e50099642b5" candidate_uuid: "19df7f5cfabb42caa62828004a614a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2b93a688c546cc923ea8df822ca21e"
I20260620 14:29:38.224475 19205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47752a898a744c687cb3e50099642b5" candidate_uuid: "19df7f5cfabb42caa62828004a614a3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63362f25483c4b0a9cf78331da1f9674"
I20260620 14:29:38.224541 19336 raft_consensus.cc:3037] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:38.224607 19205 raft_consensus.cc:3037] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:38.225648 19336 raft_consensus.cc:2454] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19df7f5cfabb42caa62828004a614a3a in term 1.
I20260620 14:29:38.225675 19205 raft_consensus.cc:2454] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19df7f5cfabb42caa62828004a614a3a in term 1.
I20260620 14:29:38.225952 19400 leader_election.cc:302] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19df7f5cfabb42caa62828004a614a3a, ce2b93a688c546cc923ea8df822ca21e; no voters:
I20260620 14:29:38.226151 19536 raft_consensus.cc:2781] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:38.226323 19536 raft_consensus.cc:686] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 1 LEADER]: Becoming Leader. State: Replica: 19df7f5cfabb42caa62828004a614a3a, State: Running, Role: LEADER
I20260620 14:29:38.226575 19536 consensus_queue.cc:237] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19df7f5cfabb42caa62828004a614a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44473 } } peers { permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 } } peers { permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 } }
I20260620 14:29:38.229343 19062 catalog_manager.cc:5495] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 19df7f5cfabb42caa62828004a614a3a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "19df7f5cfabb42caa62828004a614a3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19df7f5cfabb42caa62828004a614a3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 } health_report { overall_health: UNKNOWN } } }
I20260620 14:29:38.241057 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:38.238040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.0.98.3:44473)
0620 14:29:38.238155 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:38.238158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:38.238170 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:29:38.238290 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:38.238292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:38.238337 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:38.238416 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:29:38.238606 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.239055 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.239833 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:38.240022 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:38.240096 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:38.240316 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:29:38.240681 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:29:38.240796 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:38.240863 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:29:38.240928 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:29:38.246212 19205 raft_consensus.cc:1261] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674 [term 1 FOLLOWER]: Refusing update from remote peer 19df7f5cfabb42caa62828004a614a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:38.246321 19336 raft_consensus.cc:1261] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e [term 1 FOLLOWER]: Refusing update from remote peer 19df7f5cfabb42caa62828004a614a3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:38.246950 19540 consensus_queue.cc:1035] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce2b93a688c546cc923ea8df822ca21e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:38.247120 19536 consensus_queue.cc:1035] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63362f25483c4b0a9cf78331da1f9674" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:38.254347 19545 mvcc.cc:205] Tried to move back new op lower bound from 7298931827689869312 to 7298931827617955840. Current Snapshot: MvccSnapshot[applied={T|T < 7298931827689869312}]
I20260620 14:29:38.263522 19062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:38.263693 19062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:38.265872 19062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b843455ea9c948b9bd884a6d495dc77b: Sending DeleteTablet for 3 replicas of tablet e47752a898a744c687cb3e50099642b5
I20260620 14:29:38.266634 19185 tablet_service.cc:1514] Processing DeleteTablet for tablet e47752a898a744c687cb3e50099642b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:38 UTC) from {username='slave'} at 127.0.0.1:52840
I20260620 14:29:38.266631 19445 tablet_service.cc:1514] Processing DeleteTablet for tablet e47752a898a744c687cb3e50099642b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:38 UTC) from {username='slave'} at 127.0.0.1:40234
I20260620 14:29:38.267150 19554 tablet_replica.cc:330] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: stopping tablet replica
I20260620 14:29:38.267237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 19123
I20260620 14:29:38.267242 19553 tablet_replica.cc:330] T e47752a898a744c687cb3e50099642b5 P 63362f25483c4b0a9cf78331da1f9674: stopping tablet replica
I20260620 14:29:38.267207 19316 tablet_service.cc:1514] Processing DeleteTablet for tablet e47752a898a744c687cb3e50099642b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:38 UTC) from {username='slave'} at 127.0.0.1:44230
I20260620 14:29:38.268445 19554 raft_consensus.cc:2229] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:29:38.268824 19554 raft_consensus.cc:2258] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:38.269085 19555 tablet_replica.cc:330] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: stopping tablet replica
I20260620 14:29:38.269433 19555 raft_consensus.cc:2229] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:38.269773 19555 raft_consensus.cc:2258] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:38.270373 19554 ts_tablet_manager.cc:1905] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:38.271216 19555 ts_tablet_manager.cc:1905] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:38.273032 19554 ts_tablet_manager.cc:1918] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:38.273128 19554 log.cc:1196] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/e47752a898a744c687cb3e50099642b5
I20260620 14:29:38.273465 19554 ts_tablet_manager.cc:1939] T e47752a898a744c687cb3e50099642b5 P 19df7f5cfabb42caa62828004a614a3a: Deleting consensus metadata
I20260620 14:29:38.274166 19555 ts_tablet_manager.cc:1918] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:38.274318 19555 log.cc:1196] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/e47752a898a744c687cb3e50099642b5
I20260620 14:29:38.274678 19555 ts_tablet_manager.cc:1939] T e47752a898a744c687cb3e50099642b5 P ce2b93a688c546cc923ea8df822ca21e: Deleting consensus metadata
W20260620 14:29:38.274705 19048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42255 (error 108)
I20260620 14:29:38.274865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 19254
I20260620 14:29:38.274900 19047 catalog_manager.cc:4841] TS 19df7f5cfabb42caa62828004a614a3a (127.0.98.3:44473): tablet e47752a898a744c687cb3e50099642b5 (table test-table [id=0abb0cd9847c479a9d865e6fa4f8b9eb]) successfully deleted
I20260620 14:29:38.276119 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:38.275838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42255 (local address 127.0.0.1:52850)
0620 14:29:38.275966 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:38.276037 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:29:38.276305 19048 catalog_manager.cc:4584] TS 63362f25483c4b0a9cf78331da1f9674 (127.0.98.1:42255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e47752a898a744c687cb3e50099642b5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42255: connect: Connection refused (error 111)
I20260620 14:29:38.281570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 19385
I20260620 14:29:38.281756 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:38.281518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39205 (local address 127.0.0.1:44242)
0620 14:29:38.281644 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:38.281692 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:29:38.286974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 19032
2026-06-20T14:29:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1159 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:38 dist-test-slave-pk3j krb5kdc[19559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:38 dist-test-slave-pk3j krb5kdc[19559](info): set up 2 sockets
Jun 20 14:29:38 dist-test-slave-pk3j krb5kdc[19559](info): commencing operation
krb5kdc: starting...
W20260620 14:29:39.313059 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:39 dist-test-slave-pk3j krb5kdc[19559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:29:39.422345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35237
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:39.525225 19572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:39.525533 19572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:39.525604 19572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:39.529479 19572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:39.529561 19572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:39.529598 19572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:39.529654 19572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:39.529691 19572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:39.534696 19572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:39.535684 19572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:39.536646 19572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:39.544727 19580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:39.544724 19577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:39.544724 19578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:39.545034 19572 server_base.cc:1034] running on GCE node
I20260620 14:29:39.545601 19572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:39.546582 19572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:39.547760 19572 hybrid_clock.cc:648] HybridClock initialized: now 1781965779547754 us; error 29 us; skew 500 ppm
Jun 20 14:29:39 dist-test-slave-pk3j krb5kdc[19559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:39.550374 19572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:29:39.551156 19572 webserver.cc:460] Webserver started at http://127.0.98.62:36277/ using document root <none> and password file <none>
I20260620 14:29:39.551631 19572 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:39.551687 19572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:39.551834 19572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:39.553754 19572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "f110772da005456ebf66b634cd54149c"
format_stamp: "Formatted at 2026-06-20 14:29:39 on dist-test-slave-pk3j"
server_key: "ecedd93dd9fab50be2880a8eb1c77a25"
server_key_iv: "e5239e3932fb5abbd05d7b58a9ffcad0"
server_key_version: "encryptionkey@0"
I20260620 14:29:39.554252 19572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "f110772da005456ebf66b634cd54149c"
format_stamp: "Formatted at 2026-06-20 14:29:39 on dist-test-slave-pk3j"
server_key: "ecedd93dd9fab50be2880a8eb1c77a25"
server_key_iv: "e5239e3932fb5abbd05d7b58a9ffcad0"
server_key_version: "encryptionkey@0"
I20260620 14:29:39.557641 19572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:29:39.559787 19587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:39.560906 19572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:39.561028 19572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "f110772da005456ebf66b634cd54149c"
format_stamp: "Formatted at 2026-06-20 14:29:39 on dist-test-slave-pk3j"
server_key: "ecedd93dd9fab50be2880a8eb1c77a25"
server_key_iv: "e5239e3932fb5abbd05d7b58a9ffcad0"
server_key_version: "encryptionkey@0"
I20260620 14:29:39.561126 19572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:39.584659 19572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:39.587275 19572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:39.587471 19572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:39.594204 19572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35237
I20260620 14:29:39.594216 19639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35237 every 8 connection(s)
I20260620 14:29:39.595121 19572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:39.597697 19640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:39.598024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 19572
I20260620 14:29:39.598171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:39.598477 392 external_mini_cluster.cc:1428] Setting key c6c7f317f3d09f21c8a220a49bed500f
I20260620 14:29:39.604298 19640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c: Bootstrap starting.
I20260620 14:29:39.606945 19640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c: Neither blocks nor log segments found. Creating new log.
Jun 20 14:29:39 dist-test-slave-pk3j krb5kdc[19559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:39.607654 19640 log.cc:824] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:39.609380 19640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c: No bootstrap required, opened a new log
I20260620 14:29:39.612272 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:39.599797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.0.98.62:35237)
0620 14:29:39.600181 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:39.600201 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:39.600234 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:29:39.600873 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:39.600896 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:39.601190 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:39.601521 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:39.602364 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:39.603292 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:39.604427 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:39.604987 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:39.605113 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:39.607810 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:39.607815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:39.607826 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:39.607850 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:39.609683 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:39.610098 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:39.610108 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:39.610112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:39.610211 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:39.610487 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:39.610492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:39.610494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:39.610870 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:39.611064 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:39.611365 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 14:29:39.611626 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":139,"threads_started":1}
I20260620 14:29:39.612756 19640 raft_consensus.cc:348] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f110772da005456ebf66b634cd54149c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35237 } }
I20260620 14:29:39.612982 19640 raft_consensus.cc:374] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:39.613051 19640 raft_consensus.cc:729] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f110772da005456ebf66b634cd54149c, State: Initialized, Role: FOLLOWER
I20260620 14:29:39.613502 19640 consensus_queue.cc:260] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f110772da005456ebf66b634cd54149c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35237 } }
I20260620 14:29:39.613649 19640 raft_consensus.cc:388] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:39.613724 19640 raft_consensus.cc:482] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:39.613844 19640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:39.614746 19640 raft_consensus.cc:504] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f110772da005456ebf66b634cd54149c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35237 } }
I20260620 14:29:39.615057 19640 leader_election.cc:302] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f110772da005456ebf66b634cd54149c; no voters:
I20260620 14:29:39.615321 19640 leader_election.cc:288] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:39.615456 19645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:39.615662 19645 raft_consensus.cc:686] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [term 1 LEADER]: Becoming Leader. State: Replica: f110772da005456ebf66b634cd54149c, State: Running, Role: LEADER
I20260620 14:29:39.615950 19645 consensus_queue.cc:237] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f110772da005456ebf66b634cd54149c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35237 } }
I20260620 14:29:39.616446 19640 sys_catalog.cc:564] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:39.617497 19646 sys_catalog.cc:455] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f110772da005456ebf66b634cd54149c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f110772da005456ebf66b634cd54149c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35237 } } }
I20260620 14:29:39.617615 19646 sys_catalog.cc:458] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:39.617939 19647 sys_catalog.cc:455] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f110772da005456ebf66b634cd54149c. Latest consensus state: current_term: 1 leader_uuid: "f110772da005456ebf66b634cd54149c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f110772da005456ebf66b634cd54149c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35237 } } }
I20260620 14:29:39.618029 19647 sys_catalog.cc:458] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:39.618484 19654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:39.621455 19654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:39.626868 19654 catalog_manager.cc:1347] Generated new cluster ID: 36eb14f3522b4e3b8265630426786f87
I20260620 14:29:39.626950 19654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:39.649976 19654 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:39.651069 19654 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:39.659538 19654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c: Generated new TSK 0
I20260620 14:29:39.660219 19654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 14:29:39.688246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35237
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:39.794386 19666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:39.794657 19666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:39.794730 19666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:39.798849 19666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:39.798928 19666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:39.799010 19666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:29:39.803932 19666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:39.804911 19666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:39.805997 19666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:39.815310 19671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:39.815474 19666 server_base.cc:1034] running on GCE node
W20260620 14:29:39.815294 19674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:39.815294 19672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:39.816038 19666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:39.816556 19666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:39.817785 19666 hybrid_clock.cc:648] HybridClock initialized: now 1781965779817755 us; error 45 us; skew 500 ppm
Jun 20 14:29:39 dist-test-slave-pk3j krb5kdc[19559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:39.820466 19666 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 14:29:39.821321 19666 webserver.cc:460] Webserver started at http://127.0.98.1:46611/ using document root <none> and password file <none>
I20260620 14:29:39.821861 19666 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:39.821949 19666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:39.822167 19666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:39.824107 19666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "7ec5ad4cf2e342efa861f571cf5f7a92"
format_stamp: "Formatted at 2026-06-20 14:29:39 on dist-test-slave-pk3j"
server_key: "c10f642619d428cc6837ea61a82a1b8a"
server_key_iv: "976f77c3e4ec0687804820fa1f68ce6b"
server_key_version: "encryptionkey@0"
I20260620 14:29:39.824667 19666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7ec5ad4cf2e342efa861f571cf5f7a92"
format_stamp: "Formatted at 2026-06-20 14:29:39 on dist-test-slave-pk3j"
server_key: "c10f642619d428cc6837ea61a82a1b8a"
server_key_iv: "976f77c3e4ec0687804820fa1f68ce6b"
server_key_version: "encryptionkey@0"
I20260620 14:29:39.828177 19666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 14:29:39.830462 19681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:39.831621 19666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:29:39.831754 19666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "7ec5ad4cf2e342efa861f571cf5f7a92"
format_stamp: "Formatted at 2026-06-20 14:29:39 on dist-test-slave-pk3j"
server_key: "c10f642619d428cc6837ea61a82a1b8a"
server_key_iv: "976f77c3e4ec0687804820fa1f68ce6b"
server_key_version: "encryptionkey@0"
I20260620 14:29:39.831861 19666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:39.854826 19666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:39.857831 19666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:39.858024 19666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:39.858597 19666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:39.859408 19666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:39.859477 19666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:39.859543 19666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:39.859591 19666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:39.868314 19666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37051
I20260620 14:29:39.868340 19794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37051 every 8 connection(s)
I20260620 14:29:39.869174 19666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:29:39.873711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 19666
I20260620 14:29:39.873853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:29:39.874140 392 external_mini_cluster.cc:1428] Setting key eb254e0c33fe02e6421dc04b820031a0
Jun 20 14:29:39 dist-test-slave-pk3j krb5kdc[19559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:39.882225 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:39.870934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45007 (local address 127.0.98.62:35237)
0620 14:29:39.871070 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:39.871074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:39.871877 (+ 803us) server_negotiation.cc:407] Connection header received
0620 14:29:39.872756 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:39.872759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:39.872815 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:39.872886 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:39.874111 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:39.874752 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:39.875924 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:39.876159 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:39.876273 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:39.878791 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:39.878799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:39.878801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:39.878823 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:39.880247 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:39.880745 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:39.880750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:39.880752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:39.880829 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:39.881165 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:39.881169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:39.881170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:39.881391 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:39.881490 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:39.881944 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 14:29:39.882058 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 14:29:39.883015 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:39.871293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35237 (local address 127.0.98.1:45007)
0620 14:29:39.871721 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:39.871761 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:39.872528 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:39.873044 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:39.873050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:39.873443 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:39.873979 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:39.873994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:39.874867 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:39.874870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:39.875808 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:39.875818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:39.876305 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:39.876309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:39.876468 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:39.877029 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:39.877044 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:39.878631 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:39.880374 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:39.880379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:39.880394 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:39.880639 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:39.880917 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:39.880922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:39.880924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:39.881066 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:39.881488 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:39.881494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:39.881722 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 14:29:39.881895 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 14:29:39.882197 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260620 14:29:39.884202 19795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35237
I20260620 14:29:39.884455 19795 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 14:29:39.884994 19795 heartbeater.cc:511] Master 127.0.98.62:35237 requested a full tablet report, sending...
I20260620 14:29:39.886504 19604 ts_manager.cc:194] Registered new tserver with Master: 7ec5ad4cf2e342efa861f571cf5f7a92 (127.0.98.1:37051)
I20260620 14:29:39.887619 19604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 14:29:39.897859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35237
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:40.007054 19800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:40.007282 19800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:40.007323 19800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:40.011417 19800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:40.011497 19800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:40.011588 19800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:29:40.016696 19800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:40.017745 19800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:40.018766 19800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:40.027724 19806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:40.027776 19808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:40.027726 19805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:40.028013 19800 server_base.cc:1034] running on GCE node
I20260620 14:29:40.028522 19800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:40.029107 19800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:40.030336 19800 hybrid_clock.cc:648] HybridClock initialized: now 1781965780030319 us; error 33 us; skew 500 ppm
Jun 20 14:29:40 dist-test-slave-pk3j krb5kdc[19559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:40.032958 19800 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 14:29:40.033850 19800 webserver.cc:460] Webserver started at http://127.0.98.2:42269/ using document root <none> and password file <none>
I20260620 14:29:40.034368 19800 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:40.034423 19800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:40.034577 19800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:40.036494 19800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418"
format_stamp: "Formatted at 2026-06-20 14:29:40 on dist-test-slave-pk3j"
server_key: "dda97c2f0a7125bfae6d13a9f1cdd261"
server_key_iv: "2ddcfff8b8f197e447939b5f3f188d2f"
server_key_version: "encryptionkey@0"
I20260620 14:29:40.036981 19800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418"
format_stamp: "Formatted at 2026-06-20 14:29:40 on dist-test-slave-pk3j"
server_key: "dda97c2f0a7125bfae6d13a9f1cdd261"
server_key_iv: "2ddcfff8b8f197e447939b5f3f188d2f"
server_key_version: "encryptionkey@0"
I20260620 14:29:40.040395 19800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:29:40.042618 19815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:40.043954 19800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:29:40.044075 19800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418"
format_stamp: "Formatted at 2026-06-20 14:29:40 on dist-test-slave-pk3j"
server_key: "dda97c2f0a7125bfae6d13a9f1cdd261"
server_key_iv: "2ddcfff8b8f197e447939b5f3f188d2f"
server_key_version: "encryptionkey@0"
I20260620 14:29:40.044176 19800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:40.061036 19800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:40.063668 19800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:40.063869 19800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:40.064396 19800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:40.065147 19800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:40.065194 19800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:40.065317 19800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:40.065357 19800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:40.074163 19800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32801
I20260620 14:29:40.074196 19928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32801 every 8 connection(s)
I20260620 14:29:40.074955 19800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:29:40.083065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 19800
I20260620 14:29:40.083227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
Jun 20 14:29:40 dist-test-slave-pk3j krb5kdc[19559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:40.083561 392 external_mini_cluster.cc:1428] Setting key f7835605205b0f9584473983dbe7f84b
I20260620 14:29:40.087815 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.076555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53801 (local address 127.0.98.62:35237)
0620 14:29:40.076683 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:40.076687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:40.077449 (+ 762us) server_negotiation.cc:407] Connection header received
0620 14:29:40.078302 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:40.078306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:40.078362 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:40.078468 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:40.079540 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.080075 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.081188 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.081448 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.081557 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.083915 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:40.083918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:40.083920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:40.083966 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:40.085635 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:40.086241 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:40.086244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:40.086245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:40.086311 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:40.086710 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:40.086716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:40.086719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:40.086947 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:40.087055 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:40.087492 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 14:29:40.087611 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:29:40.088603 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.076842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35237 (local address 127.0.98.2:53801)
0620 14:29:40.077311 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:40.077350 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:40.078106 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:40.078632 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:40.078639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:40.078964 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:40.079397 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.079410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.080212 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.080215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.081060 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.081069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.081592 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.081596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.081751 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.082256 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:40.082271 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:40.083757 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:40.085752 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:40.085762 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:40.085778 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:40.086139 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:40.086395 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:40.086402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:40.086407 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:40.086600 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:40.087052 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:40.087057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:40.087265 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 14:29:40.087411 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 14:29:40.087626 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260620 14:29:40.089848 19929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35237
I20260620 14:29:40.090067 19929 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:40.090497 19929 heartbeater.cc:511] Master 127.0.98.62:35237 requested a full tablet report, sending...
I20260620 14:29:40.091531 19604 ts_manager.cc:194] Registered new tserver with Master: d13c1b0af5b04eaa9992bc4ccb8d7418 (127.0.98.2:32801)
I20260620 14:29:40.092331 19604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53801
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 14:29:40.109148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35237
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:40.212985 19934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:40.213255 19934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:40.213313 19934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:40.217264 19934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:40.217309 19934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:40.217362 19934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:29:40.222225 19934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:40.223039 19934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:40.223965 19934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:40.232652 19940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:40.232671 19939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:40.232890 19942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:40.232937 19934 server_base.cc:1034] running on GCE node
I20260620 14:29:40.233387 19934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:40.233897 19934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:40.235064 19934 hybrid_clock.cc:648] HybridClock initialized: now 1781965780235035 us; error 39 us; skew 500 ppm
Jun 20 14:29:40 dist-test-slave-pk3j krb5kdc[19559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:40.237725 19934 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 14:29:40.238512 19934 webserver.cc:460] Webserver started at http://127.0.98.3:44165/ using document root <none> and password file <none>
I20260620 14:29:40.239024 19934 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:40.239115 19934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:40.239326 19934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:40.241150 19934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "59500ecd48d54d95b601bfa2b4203476"
format_stamp: "Formatted at 2026-06-20 14:29:40 on dist-test-slave-pk3j"
server_key: "b3a3d7768f6f47dbd3291c616e0412ca"
server_key_iv: "64321be19a80313499703299c619e4e5"
server_key_version: "encryptionkey@0"
I20260620 14:29:40.241724 19934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "59500ecd48d54d95b601bfa2b4203476"
format_stamp: "Formatted at 2026-06-20 14:29:40 on dist-test-slave-pk3j"
server_key: "b3a3d7768f6f47dbd3291c616e0412ca"
server_key_iv: "64321be19a80313499703299c619e4e5"
server_key_version: "encryptionkey@0"
I20260620 14:29:40.244979 19934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:29:40.247148 19949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:40.248211 19934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:40.248366 19934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "59500ecd48d54d95b601bfa2b4203476"
format_stamp: "Formatted at 2026-06-20 14:29:40 on dist-test-slave-pk3j"
server_key: "b3a3d7768f6f47dbd3291c616e0412ca"
server_key_iv: "64321be19a80313499703299c619e4e5"
server_key_version: "encryptionkey@0"
I20260620 14:29:40.248478 19934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:40.268257 19934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:40.271370 19934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:40.271559 19934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:40.272099 19934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:40.272843 19934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:40.272920 19934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:40.273000 19934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:40.273049 19934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:40.281378 19934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39059
I20260620 14:29:40.281404 20062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39059 every 8 connection(s)
I20260620 14:29:40.282161 19934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:29:40.284101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 19934
I20260620 14:29:40.284221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:29:40.284546 392 external_mini_cluster.cc:1428] Setting key 9989fd5ca5456df1f903364b442e38e0
Jun 20 14:29:40 dist-test-slave-pk3j krb5kdc[19559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:40.294068 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.283744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32831 (local address 127.0.98.62:35237)
0620 14:29:40.283894 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:40.283900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:40.284761 (+ 861us) server_negotiation.cc:407] Connection header received
0620 14:29:40.285866 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:40.285869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:40.285923 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:40.285986 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:40.287128 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.287633 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.288631 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.288825 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.288898 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.291054 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:40.291058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:40.291059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:40.291081 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:40.292465 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:40.292858 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:40.292861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:40.292862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:40.292929 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:40.293183 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:40.293185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:40.293186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:40.293377 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:40.293458 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:40.293852 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 14:29:40.293941 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:29:40.294754 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.284098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35237 (local address 127.0.98.3:32831)
0620 14:29:40.284600 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:40.284654 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:40.285652 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:40.286160 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:40.286167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:40.286520 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:40.287011 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.287022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.287731 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.287733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.288537 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.288545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.288937 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.288940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.289067 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.289574 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:29:40.289589 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:40.290915 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:40.292552 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:40.292556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:40.292565 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:40.292781 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:40.293002 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:29:40.293004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:29:40.293005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:29:40.293112 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:29:40.293467 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:40.293471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:40.293657 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 14:29:40.293810 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 14:29:40.294036 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
I20260620 14:29:40.295737 20063 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35237
I20260620 14:29:40.295908 20063 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:40.296279 20063 heartbeater.cc:511] Master 127.0.98.62:35237 requested a full tablet report, sending...
I20260620 14:29:40.297148 19604 ts_manager.cc:194] Registered new tserver with Master: 59500ecd48d54d95b601bfa2b4203476 (127.0.98.3:39059)
I20260620 14:29:40.297690 19604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32831
I20260620 14:29:40.298178 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:29:40.306771 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.300124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.0.98.62:35237)
0620 14:29:40.300250 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:40.300254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:40.300398 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:29:40.300490 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:40.300492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:40.300522 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:40.300583 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:40.301173 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.301705 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.302462 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.302680 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.302791 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.303903 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:40.303906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:40.303909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:40.303930 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:40.305047 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:40.305513 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:40.305517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:40.305519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:40.305598 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:40.305891 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:40.305895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:40.305896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:40.306086 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:40.306223 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:40.306523 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:29:40.306624 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:29:40.309418 19603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:29:40.312177 19603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:29:40.325090 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.320695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.0.98.2:32801)
0620 14:29:40.320993 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:40.321001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:40.321038 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:29:40.321098 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:40.321101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:40.321303 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:40.321469 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:40.321741 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.322493 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.323887 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.324462 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.324581 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.324724 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:40.324853 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:29:40.324940 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":120,"threads_started":1}
I20260620 14:29:40.325085 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.320282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32801 (local address 127.0.0.1:38622)
0620 14:29:40.320699 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:40.320712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:40.320792 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:40.321460 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:40.321466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:40.321486 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:40.321613 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.321619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.322614 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.322619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.323765 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.323775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.324580 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.324583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.324692 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.324725 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:29:40.324828 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:29:40.324944 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260620 14:29:40.325476 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.319582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39059 (local address 127.0.0.1:34742)
0620 14:29:40.319992 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:40.320039 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:40.320150 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:40.321487 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:40.321490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:40.321517 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:40.321639 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.321648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.323117 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.323120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.324030 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.324039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.325054 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.325057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.325148 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.325189 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:29:40.325270 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:29:40.325345 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":11,"spinlock_wait_cycles":1280,"thread_start_us":91,"threads_started":1}
I20260620 14:29:40.326107 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.319737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.0.98.3:39059)
0620 14:29:40.321024 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:40.321036 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:40.321058 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:29:40.321125 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:40.321130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:40.321330 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:40.321777 (+ 447us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:40.321919 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.322973 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.324148 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.324924 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.325666 (+ 742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.325827 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:40.325872 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 14:29:40.325973 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":111,"threads_started":1}
I20260620 14:29:40.327728 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.319947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37051 (local address 127.0.0.1:59392)
0620 14:29:40.320521 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:40.320533 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:40.320995 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:40.321660 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:40.321665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:40.321686 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:40.321776 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.321784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.323117 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.323120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.326325 (+ 3205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.326335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.327345 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.327349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.327439 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.327458 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:29:40.327536 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 14:29:40.327601 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":31,"thread_start_us":88,"threads_started":1}
I20260620 14:29:40.327777 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.320364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.0.98.1:37051)
0620 14:29:40.321272 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:40.321283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:40.321303 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:29:40.321373 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:40.321382 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:40.321551 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:40.321686 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:40.321882 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.322967 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.326470 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.327227 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.327344 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.327472 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:40.327560 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 14:29:40.327640 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":89,"threads_started":1}
I20260620 14:29:40.328177 19863 tablet_service.cc:1467] Processing CreateTablet for tablet d8aa129a86974c8bba3ecab987db01f5 (DEFAULT_TABLE table=test-table [id=6bf8e1f3ce924786ab9e26a5df5be551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:40.328698 19997 tablet_service.cc:1467] Processing CreateTablet for tablet d8aa129a86974c8bba3ecab987db01f5 (DEFAULT_TABLE table=test-table [id=6bf8e1f3ce924786ab9e26a5df5be551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:40.329298 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8aa129a86974c8bba3ecab987db01f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:40.329660 19997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8aa129a86974c8bba3ecab987db01f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:40.330184 19729 tablet_service.cc:1467] Processing CreateTablet for tablet d8aa129a86974c8bba3ecab987db01f5 (DEFAULT_TABLE table=test-table [id=6bf8e1f3ce924786ab9e26a5df5be551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:40.330965 19729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8aa129a86974c8bba3ecab987db01f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:40.336377 20080 tablet_bootstrap.cc:492] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92: Bootstrap starting.
I20260620 14:29:40.336933 20081 tablet_bootstrap.cc:492] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: Bootstrap starting.
I20260620 14:29:40.337468 20082 tablet_bootstrap.cc:492] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: Bootstrap starting.
I20260620 14:29:40.338713 20080 tablet_bootstrap.cc:654] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:40.339298 20082 tablet_bootstrap.cc:654] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:40.339720 20080 log.cc:824] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:40.340301 20082 log.cc:824] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:40.341482 20080 tablet_bootstrap.cc:492] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92: No bootstrap required, opened a new log
I20260620 14:29:40.341676 20080 ts_tablet_manager.cc:1397] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:29:40.342175 20082 tablet_bootstrap.cc:492] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: No bootstrap required, opened a new log
I20260620 14:29:40.342347 20082 ts_tablet_manager.cc:1397] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:29:40.342352 20081 tablet_bootstrap.cc:654] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:40.343422 20081 log.cc:824] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:40.344717 20080 raft_consensus.cc:348] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59500ecd48d54d95b601bfa2b4203476" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 } }
I20260620 14:29:40.344933 20080 raft_consensus.cc:374] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:40.345010 20080 raft_consensus.cc:729] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec5ad4cf2e342efa861f571cf5f7a92, State: Initialized, Role: FOLLOWER
I20260620 14:29:40.345521 20080 consensus_queue.cc:260] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59500ecd48d54d95b601bfa2b4203476" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 } }
I20260620 14:29:40.345963 20082 raft_consensus.cc:348] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59500ecd48d54d95b601bfa2b4203476" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 } }
I20260620 14:29:40.346174 19795 heartbeater.cc:503] Master 127.0.98.62:35237 was elected leader, sending a full tablet report...
I20260620 14:29:40.346206 20082 raft_consensus.cc:374] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:40.346256 20082 raft_consensus.cc:729] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13c1b0af5b04eaa9992bc4ccb8d7418, State: Initialized, Role: FOLLOWER
I20260620 14:29:40.346532 20080 ts_tablet_manager.cc:1428] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:29:40.346639 20082 consensus_queue.cc:260] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59500ecd48d54d95b601bfa2b4203476" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 } }
I20260620 14:29:40.347257 19929 heartbeater.cc:503] Master 127.0.98.62:35237 was elected leader, sending a full tablet report...
I20260620 14:29:40.347558 20082 ts_tablet_manager.cc:1428] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:29:40.347899 20081 tablet_bootstrap.cc:492] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: No bootstrap required, opened a new log
I20260620 14:29:40.348079 20081 ts_tablet_manager.cc:1397] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260620 14:29:40.351421 20081 raft_consensus.cc:348] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59500ecd48d54d95b601bfa2b4203476" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 } }
I20260620 14:29:40.351613 20081 raft_consensus.cc:374] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:40.351652 20081 raft_consensus.cc:729] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59500ecd48d54d95b601bfa2b4203476, State: Initialized, Role: FOLLOWER
I20260620 14:29:40.351959 20081 consensus_queue.cc:260] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59500ecd48d54d95b601bfa2b4203476" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 } }
I20260620 14:29:40.352502 20063 heartbeater.cc:503] Master 127.0.98.62:35237 was elected leader, sending a full tablet report...
I20260620 14:29:40.352783 20081 ts_tablet_manager.cc:1428] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 14:29:40.370270 19796 tablet.cc:2369] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:29:40.533154 20064 tablet.cc:2369] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:29:40.576097 19930 tablet.cc:2369] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:29:40.603873 20088 raft_consensus.cc:482] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:40.604106 20088 raft_consensus.cc:504] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59500ecd48d54d95b601bfa2b4203476" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 } }
I20260620 14:29:40.605368 20088 leader_election.cc:288] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13c1b0af5b04eaa9992bc4ccb8d7418 (127.0.98.2:32801), 7ec5ad4cf2e342efa861f571cf5f7a92 (127.0.98.1:37051)
I20260620 14:29:40.608872 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.605601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32801 (local address 127.0.98.3:60107)
0620 14:29:40.605809 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:40.605825 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:40.605928 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:40.606165 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:40.606168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:40.606182 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:40.606294 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.606299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.607006 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.607010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.607822 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.607831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.608454 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.608458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.608558 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.608586 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:29:40.608673 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:29:40.608738 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":27}
I20260620 14:29:40.608994 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.605748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60107 (local address 127.0.98.2:32801)
0620 14:29:40.605941 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:40.605945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:40.605962 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:29:40.606038 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:40.606040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:40.606077 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:40.606156 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:40.606412 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.606889 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.607955 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.608333 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.608455 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.608518 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:40.608790 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:29:40.608878 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 14:29:40.609102 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.605748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44743 (local address 127.0.98.1:37051)
0620 14:29:40.605876 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:40.605881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:40.606052 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:29:40.606207 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:40.606214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:40.606251 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:40.606345 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:40.606591 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.607170 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.608165 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.608548 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.608661 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.608720 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:40.608903 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:29:40.608970 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:29:40.609486 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.605601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37051 (local address 127.0.98.3:44743)
0620 14:29:40.605948 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:40.605959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:40.606089 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:40.606353 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:40.606356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:40.606370 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:40.606483 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.606488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.607269 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.607272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.608044 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:40.608052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.608660 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.608664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:40.608775 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.608795 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:29:40.609248 (+ 453us) client_negotiation.cc:240] Negotiation successful
0620 14:29:40.609316 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":52,"thread_start_us":99,"threads_started":1}
I20260620 14:29:40.609642 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa129a86974c8bba3ecab987db01f5" candidate_uuid: "59500ecd48d54d95b601bfa2b4203476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" is_pre_election: true
I20260620 14:29:40.609965 19883 raft_consensus.cc:2454] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59500ecd48d54d95b601bfa2b4203476 in term 0.
I20260620 14:29:40.610163 19749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa129a86974c8bba3ecab987db01f5" candidate_uuid: "59500ecd48d54d95b601bfa2b4203476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" is_pre_election: true
I20260620 14:29:40.610445 19749 raft_consensus.cc:2454] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59500ecd48d54d95b601bfa2b4203476 in term 0.
I20260620 14:29:40.610460 19950 leader_election.cc:302] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59500ecd48d54d95b601bfa2b4203476, d13c1b0af5b04eaa9992bc4ccb8d7418; no voters:
I20260620 14:29:40.610746 20088 raft_consensus.cc:2781] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:29:40.610846 20088 raft_consensus.cc:482] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:29:40.610908 20088 raft_consensus.cc:3037] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:40.611889 20088 raft_consensus.cc:504] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59500ecd48d54d95b601bfa2b4203476" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 } }
I20260620 14:29:40.612214 20088 leader_election.cc:288] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [CANDIDATE]: Term 1 election: Requested vote from peers d13c1b0af5b04eaa9992bc4ccb8d7418 (127.0.98.2:32801), 7ec5ad4cf2e342efa861f571cf5f7a92 (127.0.98.1:37051)
I20260620 14:29:40.612576 19749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa129a86974c8bba3ecab987db01f5" candidate_uuid: "59500ecd48d54d95b601bfa2b4203476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92"
I20260620 14:29:40.612696 19749 raft_consensus.cc:3037] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:40.613332 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8aa129a86974c8bba3ecab987db01f5" candidate_uuid: "59500ecd48d54d95b601bfa2b4203476" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418"
I20260620 14:29:40.613432 19883 raft_consensus.cc:3037] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:40.613817 19749 raft_consensus.cc:2454] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59500ecd48d54d95b601bfa2b4203476 in term 1.
I20260620 14:29:40.614169 19953 leader_election.cc:302] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59500ecd48d54d95b601bfa2b4203476, 7ec5ad4cf2e342efa861f571cf5f7a92; no voters:
I20260620 14:29:40.614326 20088 raft_consensus.cc:2781] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:40.614560 20088 raft_consensus.cc:686] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 1 LEADER]: Becoming Leader. State: Replica: 59500ecd48d54d95b601bfa2b4203476, State: Running, Role: LEADER
I20260620 14:29:40.614773 19883 raft_consensus.cc:2454] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59500ecd48d54d95b601bfa2b4203476 in term 1.
I20260620 14:29:40.614890 20088 consensus_queue.cc:237] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59500ecd48d54d95b601bfa2b4203476" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } } peers { permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } } peers { permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 } }
I20260620 14:29:40.617945 19603 catalog_manager.cc:5495] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59500ecd48d54d95b601bfa2b4203476 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "59500ecd48d54d95b601bfa2b4203476" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59500ecd48d54d95b601bfa2b4203476" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 } health_report { overall_health: UNKNOWN } } }
I20260620 14:29:40.688182 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.684992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.0.98.3:39059)
0620 14:29:40.685122 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:40.685126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:40.685156 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:29:40.685257 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:40.685259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:40.685314 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:40.685392 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:29:40.685604 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.686093 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.686931 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.687135 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.687214 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.687456 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:29:40.687858 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:29:40.687947 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:40.688003 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:29:40.688050 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:29:40.693852 19883 raft_consensus.cc:1261] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418 [term 1 FOLLOWER]: Refusing update from remote peer 59500ecd48d54d95b601bfa2b4203476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:40.693851 19749 raft_consensus.cc:1261] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92 [term 1 FOLLOWER]: Refusing update from remote peer 59500ecd48d54d95b601bfa2b4203476: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:40.694555 20090 consensus_queue.cc:1035] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13c1b0af5b04eaa9992bc4ccb8d7418" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:40.694727 20088 consensus_queue.cc:1035] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec5ad4cf2e342efa861f571cf5f7a92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:40.704118 20094 mvcc.cc:205] Tried to move back new op lower bound from 7298931837714731008 to 7298931837400817664. Current Snapshot: MvccSnapshot[applied={T|T < 7298931837714731008}]
I20260620 14:29:40.710579 19604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:40.710763 19604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:40.712909 19604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f110772da005456ebf66b634cd54149c: Sending DeleteTablet for 3 replicas of tablet d8aa129a86974c8bba3ecab987db01f5
I20260620 14:29:40.713680 19863 tablet_service.cc:1514] Processing DeleteTablet for tablet d8aa129a86974c8bba3ecab987db01f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38622
I20260620 14:29:40.713719 19997 tablet_service.cc:1514] Processing DeleteTablet for tablet d8aa129a86974c8bba3ecab987db01f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34742
I20260620 14:29:40.713701 19729 tablet_service.cc:1514] Processing DeleteTablet for tablet d8aa129a86974c8bba3ecab987db01f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59392
I20260620 14:29:40.714172 20103 tablet_replica.cc:330] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92: stopping tablet replica
I20260620 14:29:40.714453 20103 raft_consensus.cc:2229] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:40.714731 20103 raft_consensus.cc:2258] T d8aa129a86974c8bba3ecab987db01f5 P 7ec5ad4cf2e342efa861f571cf5f7a92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:40.714776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 19666
I20260620 14:29:40.715065 20104 tablet_replica.cc:330] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: stopping tablet replica
I20260620 14:29:40.715354 20102 tablet_replica.cc:330] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: stopping tablet replica
I20260620 14:29:40.715587 20102 raft_consensus.cc:2229] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:40.715832 20102 raft_consensus.cc:2258] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:40.716173 20104 raft_consensus.cc:2229] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:29:40.716460 20104 raft_consensus.cc:2258] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:40.718632 20102 ts_tablet_manager.cc:1905] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:40.718719 20104 ts_tablet_manager.cc:1905] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:40.721333 20104 ts_tablet_manager.cc:1918] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:40.721422 20104 log.cc:1196] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/d8aa129a86974c8bba3ecab987db01f5
I20260620 14:29:40.721773 20104 ts_tablet_manager.cc:1939] T d8aa129a86974c8bba3ecab987db01f5 P 59500ecd48d54d95b601bfa2b4203476: Deleting consensus metadata
I20260620 14:29:40.722203 20102 ts_tablet_manager.cc:1918] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:40.722352 20102 log.cc:1196] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/d8aa129a86974c8bba3ecab987db01f5
I20260620 14:29:40.722718 20102 ts_tablet_manager.cc:1939] T d8aa129a86974c8bba3ecab987db01f5 P d13c1b0af5b04eaa9992bc4ccb8d7418: Deleting consensus metadata
I20260620 14:29:40.722851 19588 catalog_manager.cc:4841] TS 59500ecd48d54d95b601bfa2b4203476 (127.0.98.3:39059): tablet d8aa129a86974c8bba3ecab987db01f5 (table test-table [id=6bf8e1f3ce924786ab9e26a5df5be551]) successfully deleted
W20260620 14:29:40.723151 19591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37051 (error 108)
I20260620 14:29:40.724118 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.723826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37051 (local address 127.0.0.1:59396)
0620 14:29:40.723958 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:40.724041 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:29:40.724223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 19800
I20260620 14:29:40.724352 19588 catalog_manager.cc:4841] TS d13c1b0af5b04eaa9992bc4ccb8d7418 (127.0.98.2:32801): tablet d8aa129a86974c8bba3ecab987db01f5 (table test-table [id=6bf8e1f3ce924786ab9e26a5df5be551]) successfully deleted
W20260620 14:29:40.724332 19591 catalog_manager.cc:4584] TS 7ec5ad4cf2e342efa861f571cf5f7a92 (127.0.98.1:37051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8aa129a86974c8bba3ecab987db01f5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37051: connect: Connection refused (error 111)
I20260620 14:29:40.729637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 19934
I20260620 14:29:40.735297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 19572
2026-06-20T14:29:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2447 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:29:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:40Z Disabled control of system clock
I20260620 14:29:40.763191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44325
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:29:40.868037 20109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:40.868268 20109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:40.868300 20109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:40.872087 20109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:40.872133 20109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:40.872148 20109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:40.872164 20109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:40.872179 20109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:40.876641 20109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:40.877588 20109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:40.878775 20109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:40.886708 20115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:40.886708 20114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:40.886755 20117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:40.886987 20109 server_base.cc:1034] running on GCE node
I20260620 14:29:40.887619 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:40.888581 20109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:40.889756 20109 hybrid_clock.cc:648] HybridClock initialized: now 1781965780889734 us; error 45 us; skew 500 ppm
I20260620 14:29:40.891479 20109 webserver.cc:460] Webserver started at http://127.0.98.62:46709/ using document root <none> and password file <none>
I20260620 14:29:40.892014 20109 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:40.892097 20109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:40.892304 20109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:40.894248 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "d559cae7d3a6474a8504e268d7e02fa3"
format_stamp: "Formatted at 2026-06-20 14:29:40 on dist-test-slave-pk3j"
server_key: "778b2281fe37ef247e43ed99add1e829"
server_key_iv: "e6505514631fb07b33e35aea78523c65"
server_key_version: "encryptionkey@0"
I20260620 14:29:40.894794 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "d559cae7d3a6474a8504e268d7e02fa3"
format_stamp: "Formatted at 2026-06-20 14:29:40 on dist-test-slave-pk3j"
server_key: "778b2281fe37ef247e43ed99add1e829"
server_key_iv: "e6505514631fb07b33e35aea78523c65"
server_key_version: "encryptionkey@0"
I20260620 14:29:40.898195 20109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 14:29:40.900355 20123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:40.901556 20109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:40.901684 20109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "d559cae7d3a6474a8504e268d7e02fa3"
format_stamp: "Formatted at 2026-06-20 14:29:40 on dist-test-slave-pk3j"
server_key: "778b2281fe37ef247e43ed99add1e829"
server_key_iv: "e6505514631fb07b33e35aea78523c65"
server_key_version: "encryptionkey@0"
I20260620 14:29:40.901785 20109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:40.932210 20109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:40.932921 20109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:40.933122 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:40.940044 20109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44325
I20260620 14:29:40.940052 20175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44325 every 8 connection(s)
I20260620 14:29:40.940944 20109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:40.943882 20176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:40.948148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 20109
I20260620 14:29:40.948267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:40.948575 392 external_mini_cluster.cc:1428] Setting key 5da108abd41dc50e5469c7b387fbc203
I20260620 14:29:40.949553 20176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3: Bootstrap starting.
I20260620 14:29:40.951807 20176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:40.952518 20176 log.cc:824] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:40.954306 20176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3: No bootstrap required, opened a new log
I20260620 14:29:40.957583 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:40.950043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.0.98.62:44325)
0620 14:29:40.950651 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:40.950679 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:40.950713 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:29:40.951307 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:40.951323 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:40.951695 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:40.952030 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:40.952435 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.953504 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.954369 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:40.954891 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:40.955060 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:40.955944 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:40.955949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:40.955961 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:40.956316 (+ 355us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:40.956365 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:40.956383 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:40.956492 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:40.956606 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:29:40.956841 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
I20260620 14:29:40.957921 20176 raft_consensus.cc:348] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d559cae7d3a6474a8504e268d7e02fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44325 } }
I20260620 14:29:40.958161 20176 raft_consensus.cc:374] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:40.958240 20176 raft_consensus.cc:729] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d559cae7d3a6474a8504e268d7e02fa3, State: Initialized, Role: FOLLOWER
I20260620 14:29:40.958710 20176 consensus_queue.cc:260] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d559cae7d3a6474a8504e268d7e02fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44325 } }
I20260620 14:29:40.958871 20176 raft_consensus.cc:388] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:40.958935 20176 raft_consensus.cc:482] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:40.959025 20176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:40.960047 20176 raft_consensus.cc:504] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d559cae7d3a6474a8504e268d7e02fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44325 } }
I20260620 14:29:40.960322 20176 leader_election.cc:302] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d559cae7d3a6474a8504e268d7e02fa3; no voters:
I20260620 14:29:40.960546 20176 leader_election.cc:288] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:40.960700 20181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:40.960935 20181 raft_consensus.cc:686] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [term 1 LEADER]: Becoming Leader. State: Replica: d559cae7d3a6474a8504e268d7e02fa3, State: Running, Role: LEADER
I20260620 14:29:40.961258 20181 consensus_queue.cc:237] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d559cae7d3a6474a8504e268d7e02fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44325 } }
I20260620 14:29:40.961760 20176 sys_catalog.cc:564] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:40.962822 20182 sys_catalog.cc:455] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d559cae7d3a6474a8504e268d7e02fa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d559cae7d3a6474a8504e268d7e02fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44325 } } }
I20260620 14:29:40.962951 20182 sys_catalog.cc:458] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:40.963326 20190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:40.963308 20183 sys_catalog.cc:455] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d559cae7d3a6474a8504e268d7e02fa3. Latest consensus state: current_term: 1 leader_uuid: "d559cae7d3a6474a8504e268d7e02fa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d559cae7d3a6474a8504e268d7e02fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44325 } } }
I20260620 14:29:40.963984 20183 sys_catalog.cc:458] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:40.966470 20190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:40.971446 20190 catalog_manager.cc:1347] Generated new cluster ID: f593269d03bb4741a8a5f3da48b91e6e
I20260620 14:29:40.971506 20190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:40.979542 20190 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:40.980643 20190 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:40.991726 20190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3: Generated new TSK 0
I20260620 14:29:40.992419 20190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:40.997943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44325
--builtin_ntp_servers=127.0.98.20:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:29:41.104316 20200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:41.104540 20200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:41.104605 20200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:41.108580 20200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:41.108634 20200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:41.108685 20200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 14:29:41.113454 20200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:41.114439 20200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:41.115764 20200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:41.124660 20205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:41.124653 20206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:41.124696 20208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:41.124917 20200 server_base.cc:1034] running on GCE node
I20260620 14:29:41.125437 20200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:41.126001 20200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:41.127156 20200 hybrid_clock.cc:648] HybridClock initialized: now 1781965781127130 us; error 37 us; skew 500 ppm
I20260620 14:29:41.128930 20200 webserver.cc:460] Webserver started at http://127.0.98.1:34135/ using document root <none> and password file <none>
I20260620 14:29:41.129544 20200 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:41.129606 20200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:41.129828 20200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:41.131747 20200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/data/instance:
uuid: "63284f059b8f4622bfce7c4d52a64df5"
format_stamp: "Formatted at 2026-06-20 14:29:41 on dist-test-slave-pk3j"
server_key: "5e8fca1914e25db0498be756c7c85b64"
server_key_iv: "4a5d119f1c36bcac7adc4710119084a4"
server_key_version: "encryptionkey@0"
I20260620 14:29:41.132272 20200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/wal/instance:
uuid: "63284f059b8f4622bfce7c4d52a64df5"
format_stamp: "Formatted at 2026-06-20 14:29:41 on dist-test-slave-pk3j"
server_key: "5e8fca1914e25db0498be756c7c85b64"
server_key_iv: "4a5d119f1c36bcac7adc4710119084a4"
server_key_version: "encryptionkey@0"
I20260620 14:29:41.135644 20200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:29:41.137933 20214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:41.139034 20200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:29:41.139161 20200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/wal
uuid: "63284f059b8f4622bfce7c4d52a64df5"
format_stamp: "Formatted at 2026-06-20 14:29:41 on dist-test-slave-pk3j"
server_key: "5e8fca1914e25db0498be756c7c85b64"
server_key_iv: "4a5d119f1c36bcac7adc4710119084a4"
server_key_version: "encryptionkey@0"
I20260620 14:29:41.139261 20200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:41.158711 20200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:41.159349 20200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:41.159547 20200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:41.160101 20200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:41.161006 20200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:41.161079 20200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:41.161144 20200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:41.161191 20200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:41.169660 20200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32815
I20260620 14:29:41.169697 20327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32815 every 8 connection(s)
I20260620 14:29:41.170547 20200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/data/info.pb
I20260620 14:29:41.174074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 20200
I20260620 14:29:41.174212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-0/wal/instance
I20260620 14:29:41.174492 392 external_mini_cluster.cc:1428] Setting key 74a5e0333ec8779a63a1cd7cede2714e
I20260620 14:29:41.176455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44325
--builtin_ntp_servers=127.0.98.20:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:29:41.180158 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.172333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33619 (local address 127.0.98.62:44325)
0620 14:29:41.172480 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:41.172485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:41.173602 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 14:29:41.175280 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:41.175283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:41.175327 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:41.175408 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:41.176359 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.177145 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.178267 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.178493 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.178604 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.179689 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:41.179692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:41.179694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:41.179706 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:41.179723 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:41.179730 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:41.179807 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:41.179967 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:29:41.180027 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:29:41.181056 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.172778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44325 (local address 127.0.98.1:33619)
0620 14:29:41.173472 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:41.173512 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:41.175087 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:41.175565 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:41.175571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:41.175652 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:41.176220 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.176233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.177302 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.177304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.178130 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.178138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.178640 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.178643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.178808 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.179419 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:29:41.179420 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:41.179458 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:29:41.179517 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:41.179820 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:41.179831 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:41.179838 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:29:41.179964 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:29:41.180230 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":111,"threads_started":1}
I20260620 14:29:41.182304 20328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44325
I20260620 14:29:41.182533 20328 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:41.183003 20328 heartbeater.cc:511] Master 127.0.98.62:44325 requested a full tablet report, sending...
I20260620 14:29:41.184639 20140 ts_manager.cc:194] Registered new tserver with Master: 63284f059b8f4622bfce7c4d52a64df5 (127.0.98.1:32815)
I20260620 14:29:41.186357 20140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33619
W20260620 14:29:41.282397 20331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:41.282599 20331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:41.282630 20331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:41.286358 20331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:41.286402 20331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:41.286453 20331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 14:29:41.290771 20331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:41.291687 20331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:41.292888 20331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:41.301431 20337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:41.301493 20336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:41.301499 20339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:41.301610 20331 server_base.cc:1034] running on GCE node
I20260620 14:29:41.301978 20331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:41.302482 20331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:41.303635 20331 hybrid_clock.cc:648] HybridClock initialized: now 1781965781303622 us; error 26 us; skew 500 ppm
I20260620 14:29:41.305500 20331 webserver.cc:460] Webserver started at http://127.0.98.2:39871/ using document root <none> and password file <none>
I20260620 14:29:41.306097 20331 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:41.306156 20331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:41.306351 20331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:41.308269 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/data/instance:
uuid: "5480d22306924b3e9ac7aaf5edf9d6fe"
format_stamp: "Formatted at 2026-06-20 14:29:41 on dist-test-slave-pk3j"
server_key: "c51260cd8896205280eb7f5f3fefd1e7"
server_key_iv: "e1c8fed63f5b817ea9f657dbdbd3052b"
server_key_version: "encryptionkey@0"
I20260620 14:29:41.308789 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5480d22306924b3e9ac7aaf5edf9d6fe"
format_stamp: "Formatted at 2026-06-20 14:29:41 on dist-test-slave-pk3j"
server_key: "c51260cd8896205280eb7f5f3fefd1e7"
server_key_iv: "e1c8fed63f5b817ea9f657dbdbd3052b"
server_key_version: "encryptionkey@0"
I20260620 14:29:41.312397 20331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:29:41.314703 20345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:41.315876 20331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:29:41.315992 20331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/wal
uuid: "5480d22306924b3e9ac7aaf5edf9d6fe"
format_stamp: "Formatted at 2026-06-20 14:29:41 on dist-test-slave-pk3j"
server_key: "c51260cd8896205280eb7f5f3fefd1e7"
server_key_iv: "e1c8fed63f5b817ea9f657dbdbd3052b"
server_key_version: "encryptionkey@0"
I20260620 14:29:41.316118 20331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:41.336570 20331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:41.337172 20331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:41.337410 20331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:41.337976 20331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:41.338819 20331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:41.338865 20331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:41.338936 20331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:41.338973 20331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:41.347645 20331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45599
I20260620 14:29:41.347668 20458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45599 every 8 connection(s)
I20260620 14:29:41.348457 20331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/data/info.pb
I20260620 14:29:41.352432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 20331
I20260620 14:29:41.352553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/wal/instance
I20260620 14:29:41.352885 392 external_mini_cluster.cc:1428] Setting key ef384ae7a2bc0a78aac1557515c5fbcd
I20260620 14:29:41.354812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44325
--builtin_ntp_servers=127.0.98.20:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:29:41.358371 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.350145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54811 (local address 127.0.98.62:44325)
0620 14:29:41.350277 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:41.350281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:41.351243 (+ 962us) server_negotiation.cc:407] Connection header received
0620 14:29:41.352763 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:41.352766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:41.352812 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:41.352875 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:41.353918 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.354580 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.356300 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.356670 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.356786 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.357810 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:41.357817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:41.357823 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:41.357840 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:41.357866 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:41.357878 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:41.357966 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:41.358099 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:29:41.358168 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:29:41.359112 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.350484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44325 (local address 127.0.98.2:54811)
0620 14:29:41.351109 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:41.351144 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:41.352531 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:41.353072 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:41.353081 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:41.353179 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:41.353765 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.353778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.354714 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.354721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.356172 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.356185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.356815 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.356818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.356948 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.357571 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:29:41.357572 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:41.357607 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:29:41.357662 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:41.357970 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:41.357976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:41.357980 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:29:41.358068 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:29:41.358260 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
I20260620 14:29:41.360168 20459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44325
I20260620 14:29:41.360383 20459 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:41.360831 20459 heartbeater.cc:511] Master 127.0.98.62:44325 requested a full tablet report, sending...
I20260620 14:29:41.361892 20140 ts_manager.cc:194] Registered new tserver with Master: 5480d22306924b3e9ac7aaf5edf9d6fe (127.0.98.2:45599)
I20260620 14:29:41.362728 20140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54811
W20260620 14:29:41.460493 20462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:41.460716 20462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:41.460752 20462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:41.464924 20462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:41.464977 20462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:41.465029 20462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 14:29:41.469555 20462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:41.470423 20462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:41.471649 20462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:41.480568 20467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:41.480585 20468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:41.480561 20462 server_base.cc:1034] running on GCE node
W20260620 14:29:41.480546 20470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:41.481083 20462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:41.481652 20462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:41.482805 20462 hybrid_clock.cc:648] HybridClock initialized: now 1781965781482789 us; error 29 us; skew 500 ppm
I20260620 14:29:41.484417 20462 webserver.cc:460] Webserver started at http://127.0.98.3:38337/ using document root <none> and password file <none>
I20260620 14:29:41.484901 20462 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:41.484958 20462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:41.485106 20462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:41.486989 20462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/data/instance:
uuid: "76fffeedcdc34a34bd695005c86e18e2"
format_stamp: "Formatted at 2026-06-20 14:29:41 on dist-test-slave-pk3j"
server_key: "b37247fe02d42278bae878e649587ea0"
server_key_iv: "f94921912cad44f12af0e745563dc37e"
server_key_version: "encryptionkey@0"
I20260620 14:29:41.487464 20462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/wal/instance:
uuid: "76fffeedcdc34a34bd695005c86e18e2"
format_stamp: "Formatted at 2026-06-20 14:29:41 on dist-test-slave-pk3j"
server_key: "b37247fe02d42278bae878e649587ea0"
server_key_iv: "f94921912cad44f12af0e745563dc37e"
server_key_version: "encryptionkey@0"
I20260620 14:29:41.490833 20462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:29:41.493075 20476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:41.494247 20462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:29:41.494377 20462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/wal
uuid: "76fffeedcdc34a34bd695005c86e18e2"
format_stamp: "Formatted at 2026-06-20 14:29:41 on dist-test-slave-pk3j"
server_key: "b37247fe02d42278bae878e649587ea0"
server_key_iv: "f94921912cad44f12af0e745563dc37e"
server_key_version: "encryptionkey@0"
I20260620 14:29:41.494493 20462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:41.513979 20462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:41.514598 20462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:41.514787 20462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:41.515333 20462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:29:41.516180 20462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:29:41.516250 20462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:41.516319 20462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:29:41.516361 20462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:41.525053 20462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42291
I20260620 14:29:41.525079 20589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42291 every 8 connection(s)
I20260620 14:29:41.525940 20462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/data/info.pb
I20260620 14:29:41.530933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 20462
I20260620 14:29:41.531057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/wal/instance
I20260620 14:29:41.531325 392 external_mini_cluster.cc:1428] Setting key 99586dd428fe085290c252cc6372548a
I20260620 14:29:41.534745 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.527664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44875 (local address 127.0.98.62:44325)
0620 14:29:41.527845 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:41.527850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:41.528790 (+ 940us) server_negotiation.cc:407] Connection header received
0620 14:29:41.530091 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:41.530094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:41.530138 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:41.530210 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:41.531055 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.531761 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.533023 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.533354 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.533447 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.534367 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:41.534370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:41.534372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:41.534390 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:41.534407 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:41.534414 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:41.534471 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:41.534596 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:29:41.534641 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:29:41.535580 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.528068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44325 (local address 127.0.98.3:44875)
0620 14:29:41.528650 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:41.528685 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:41.529919 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:41.530357 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:41.530364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:41.530440 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:29:41.530906 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.530917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.531876 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.531879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.532914 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.532922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.533477 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.533479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.533609 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.534148 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:29:41.534150 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:29:41.534184 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:29:41.534241 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:29:41.534484 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:29:41.534490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:29:41.534494 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:29:41.534580 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:29:41.534766 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":108,"threads_started":1}
I20260620 14:29:41.536617 20590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44325
I20260620 14:29:41.536821 20590 heartbeater.cc:464] Registering TS with master...
I20260620 14:29:41.537283 20590 heartbeater.cc:511] Master 127.0.98.62:44325 requested a full tablet report, sending...
I20260620 14:29:41.538151 20140 ts_manager.cc:194] Registered new tserver with Master: 76fffeedcdc34a34bd695005c86e18e2 (127.0.98.3:42291)
I20260620 14:29:41.538688 20140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44875
I20260620 14:29:41.544818 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:29:41.549727 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.546652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47142 (local address 127.0.98.62:44325)
0620 14:29:41.546761 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:41.546764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:41.546946 (+ 182us) server_negotiation.cc:407] Connection header received
0620 14:29:41.547023 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:41.547025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:41.547057 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:41.547119 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:41.547337 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.547783 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.548512 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.548694 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.548762 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.549438 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:41.549441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:41.549443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:41.549452 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:41.549468 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:41.549475 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:41.549529 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:41.549598 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:29:41.549638 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:29:41.552119 20140 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:29:41.554977 20140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:29:41.567921 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.562424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42291 (local address 127.0.0.1:44458)
0620 14:29:41.562751 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:41.562799 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:41.562908 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:41.563964 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:41.563971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:41.563993 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:41.564131 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.564140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.565230 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.565234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.566507 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.566515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.567322 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.567325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.567491 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.567552 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 14:29:41.567676 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 14:29:41.567788 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":75,"thread_start_us":78,"threads_started":1}
I20260620 14:29:41.568356 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.562543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.0.98.3:42291)
0620 14:29:41.563282 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:41.563293 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:41.563330 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:29:41.563410 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:41.563415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:41.563614 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:41.563776 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:41.564269 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.565112 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.566619 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.567198 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.567321 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.567503 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:41.568115 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 14:29:41.568209 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":138,"threads_started":1}
I20260620 14:29:41.569024 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.563281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45599 (local address 127.0.0.1:59822)
0620 14:29:41.564253 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:41.564266 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:41.564357 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:41.565040 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:41.565043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:41.565060 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:41.565155 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.565163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.566228 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.566231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.567557 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.567562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.568654 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.568657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.568742 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.568765 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:29:41.568826 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:29:41.568884 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":127,"thread_start_us":85,"threads_started":1}
I20260620 14:29:41.569078 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.563767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.0.98.2:45599)
0620 14:29:41.564222 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:41.564232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:41.564335 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:29:41.564721 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:41.564726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:41.564890 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:41.565034 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:41.565337 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.566109 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.567691 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.568528 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.568640 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.568806 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:41.568853 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:29:41.568931 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":130,"threads_started":1}
I20260620 14:29:41.569355 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.563225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.0.98.1:32815)
0620 14:29:41.563878 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:41.563903 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:41.563928 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:29:41.564002 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:41.564007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:41.564314 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:41.564463 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:41.564645 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.565763 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.567672 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.568434 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.568533 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.568722 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:41.569089 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 14:29:41.569184 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":152,"threads_started":1}
I20260620 14:29:41.569566 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.562616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32815 (local address 127.0.0.1:52958)
0620 14:29:41.563498 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:41.563512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:41.563603 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:41.564441 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:41.564445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:41.564462 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:41.564552 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.564560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.565881 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.565884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.567551 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.567561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.568775 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.568778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.568862 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.568879 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:29:41.568949 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:29:41.568999 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":157,"thread_start_us":89,"threads_started":1}
I20260620 14:29:41.571548 20393 tablet_service.cc:1467] Processing CreateTablet for tablet 067d10fde3c54269a83d892f85f14297 (DEFAULT_TABLE table=test-table [id=870a8fa285fa411c81b28864f67dd990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:41.571450 20524 tablet_service.cc:1467] Processing CreateTablet for tablet 067d10fde3c54269a83d892f85f14297 (DEFAULT_TABLE table=test-table [id=870a8fa285fa411c81b28864f67dd990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:41.572365 20393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067d10fde3c54269a83d892f85f14297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:41.572433 20524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067d10fde3c54269a83d892f85f14297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:41.572160 20262 tablet_service.cc:1467] Processing CreateTablet for tablet 067d10fde3c54269a83d892f85f14297 (DEFAULT_TABLE table=test-table [id=870a8fa285fa411c81b28864f67dd990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:29:41.572966 20262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 067d10fde3c54269a83d892f85f14297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:41.577919 20607 tablet_bootstrap.cc:492] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: Bootstrap starting.
I20260620 14:29:41.579480 20609 tablet_bootstrap.cc:492] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5: Bootstrap starting.
I20260620 14:29:41.580084 20607 tablet_bootstrap.cc:654] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:41.580166 20608 tablet_bootstrap.cc:492] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: Bootstrap starting.
I20260620 14:29:41.580964 20607 log.cc:824] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:41.581557 20609 tablet_bootstrap.cc:654] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:41.582008 20608 tablet_bootstrap.cc:654] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:41.582259 20609 log.cc:824] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:41.582702 20608 log.cc:824] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:41.582947 20607 tablet_bootstrap.cc:492] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: No bootstrap required, opened a new log
I20260620 14:29:41.583137 20607 ts_tablet_manager.cc:1397] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:29:41.584021 20609 tablet_bootstrap.cc:492] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5: No bootstrap required, opened a new log
I20260620 14:29:41.584216 20609 ts_tablet_manager.cc:1397] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:29:41.584515 20608 tablet_bootstrap.cc:492] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: No bootstrap required, opened a new log
I20260620 14:29:41.584693 20608 ts_tablet_manager.cc:1397] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:29:41.586719 20607 raft_consensus.cc:348] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fffeedcdc34a34bd695005c86e18e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42291 } } peers { permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 } } peers { permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 } }
I20260620 14:29:41.586932 20607 raft_consensus.cc:374] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:41.586982 20607 raft_consensus.cc:729] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5480d22306924b3e9ac7aaf5edf9d6fe, State: Initialized, Role: FOLLOWER
I20260620 14:29:41.587355 20607 consensus_queue.cc:260] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fffeedcdc34a34bd695005c86e18e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42291 } } peers { permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 } } peers { permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 } }
I20260620 14:29:41.587936 20459 heartbeater.cc:503] Master 127.0.98.62:44325 was elected leader, sending a full tablet report...
I20260620 14:29:41.588264 20607 ts_tablet_manager.cc:1428] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:29:41.588331 20609 raft_consensus.cc:348] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fffeedcdc34a34bd695005c86e18e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42291 } } peers { permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 } } peers { permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 } }
I20260620 14:29:41.588546 20609 raft_consensus.cc:374] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:41.588599 20609 raft_consensus.cc:729] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63284f059b8f4622bfce7c4d52a64df5, State: Initialized, Role: FOLLOWER
I20260620 14:29:41.588582 20608 raft_consensus.cc:348] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fffeedcdc34a34bd695005c86e18e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42291 } } peers { permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 } } peers { permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 } }
I20260620 14:29:41.588779 20608 raft_consensus.cc:374] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:41.588829 20608 raft_consensus.cc:729] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fffeedcdc34a34bd695005c86e18e2, State: Initialized, Role: FOLLOWER
I20260620 14:29:41.588994 20609 consensus_queue.cc:260] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fffeedcdc34a34bd695005c86e18e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42291 } } peers { permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 } } peers { permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 } }
I20260620 14:29:41.589251 20608 consensus_queue.cc:260] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fffeedcdc34a34bd695005c86e18e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42291 } } peers { permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 } } peers { permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 } }
I20260620 14:29:41.589666 20328 heartbeater.cc:503] Master 127.0.98.62:44325 was elected leader, sending a full tablet report...
I20260620 14:29:41.589922 20609 ts_tablet_manager.cc:1428] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260620 14:29:41.590049 20608 ts_tablet_manager.cc:1428] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:29:41.590361 20590 heartbeater.cc:503] Master 127.0.98.62:44325 was elected leader, sending a full tablet report...
W20260620 14:29:41.599285 20460 tablet.cc:2369] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:29:41.671691 20329 tablet.cc:2369] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:29:41.776986 20591 tablet.cc:2369] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:29:41.860239 20615 raft_consensus.cc:482] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:41.860462 20615 raft_consensus.cc:504] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fffeedcdc34a34bd695005c86e18e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42291 } } peers { permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 } } peers { permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 } }
I20260620 14:29:41.861526 20615 leader_election.cc:288] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63284f059b8f4622bfce7c4d52a64df5 (127.0.98.1:32815), 5480d22306924b3e9ac7aaf5edf9d6fe (127.0.98.2:45599)
I20260620 14:29:41.865818 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.861745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32815 (local address 127.0.98.3:59681)
0620 14:29:41.862145 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:41.862161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:41.862281 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:41.862554 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:41.862556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:41.862579 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:41.862809 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.862814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.863821 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.863823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.864715 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.864732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.865399 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.865403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.865498 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.865525 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:29:41.865584 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:29:41.865635 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":86,"spinlock_wait_cycles":28160}
I20260620 14:29:41.865823 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.861916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59681 (local address 127.0.98.1:32815)
0620 14:29:41.862104 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:41.862110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:41.862235 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:29:41.862400 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:41.862402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:41.862448 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:41.862530 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:41.862925 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.863688 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.864880 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.865290 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.865397 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.865463 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:41.865627 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:29:41.865704 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:29:41.865841 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.861763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45599 (local address 127.0.98.3:44253)
0620 14:29:41.862145 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:41.862161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:29:41.862281 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:29:41.862554 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:29:41.862556 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:29:41.862578 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:29:41.862806 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.862814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.863846 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.863849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.864693 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:29:41.864701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.865453 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.865457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:29:41.865553 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.865595 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:29:41.865671 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:29:41.865726 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":105,"spinlock_wait_cycles":10752,"thread_start_us":133,"threads_started":1}
I20260620 14:29:41.865903 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.861855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44253 (local address 127.0.98.2:45599)
0620 14:29:41.861989 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:41.861994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:41.862238 (+ 244us) server_negotiation.cc:407] Connection header received
0620 14:29:41.862393 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:41.862396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:41.862448 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:41.862531 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:29:41.862946 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.863687 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.864817 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.865306 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.865421 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.865478 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:41.865704 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:29:41.865775 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:29:41.866418 20282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067d10fde3c54269a83d892f85f14297" candidate_uuid: "76fffeedcdc34a34bd695005c86e18e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63284f059b8f4622bfce7c4d52a64df5" is_pre_election: true
I20260620 14:29:41.866581 20413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067d10fde3c54269a83d892f85f14297" candidate_uuid: "76fffeedcdc34a34bd695005c86e18e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" is_pre_election: true
I20260620 14:29:41.866724 20282 raft_consensus.cc:2454] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76fffeedcdc34a34bd695005c86e18e2 in term 0.
I20260620 14:29:41.866874 20413 raft_consensus.cc:2454] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76fffeedcdc34a34bd695005c86e18e2 in term 0.
I20260620 14:29:41.867259 20479 leader_election.cc:302] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63284f059b8f4622bfce7c4d52a64df5, 76fffeedcdc34a34bd695005c86e18e2; no voters:
I20260620 14:29:41.867540 20615 raft_consensus.cc:2781] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:29:41.867621 20615 raft_consensus.cc:482] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:29:41.867661 20615 raft_consensus.cc:3037] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:41.869019 20615 raft_consensus.cc:504] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fffeedcdc34a34bd695005c86e18e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42291 } } peers { permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 } } peers { permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 } }
I20260620 14:29:41.869355 20615 leader_election.cc:288] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [CANDIDATE]: Term 1 election: Requested vote from peers 63284f059b8f4622bfce7c4d52a64df5 (127.0.98.1:32815), 5480d22306924b3e9ac7aaf5edf9d6fe (127.0.98.2:45599)
I20260620 14:29:41.869647 20282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067d10fde3c54269a83d892f85f14297" candidate_uuid: "76fffeedcdc34a34bd695005c86e18e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63284f059b8f4622bfce7c4d52a64df5"
I20260620 14:29:41.869721 20413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "067d10fde3c54269a83d892f85f14297" candidate_uuid: "76fffeedcdc34a34bd695005c86e18e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe"
I20260620 14:29:41.869783 20282 raft_consensus.cc:3037] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:41.869817 20413 raft_consensus.cc:3037] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:41.870987 20413 raft_consensus.cc:2454] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76fffeedcdc34a34bd695005c86e18e2 in term 1.
I20260620 14:29:41.870987 20282 raft_consensus.cc:2454] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76fffeedcdc34a34bd695005c86e18e2 in term 1.
I20260620 14:29:41.871275 20478 leader_election.cc:302] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5480d22306924b3e9ac7aaf5edf9d6fe, 76fffeedcdc34a34bd695005c86e18e2; no voters:
I20260620 14:29:41.871446 20615 raft_consensus.cc:2781] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:41.871670 20615 raft_consensus.cc:686] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 1 LEADER]: Becoming Leader. State: Replica: 76fffeedcdc34a34bd695005c86e18e2, State: Running, Role: LEADER
I20260620 14:29:41.871973 20615 consensus_queue.cc:237] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fffeedcdc34a34bd695005c86e18e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42291 } } peers { permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 } } peers { permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 } }
I20260620 14:29:41.874945 20139 catalog_manager.cc:5495] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76fffeedcdc34a34bd695005c86e18e2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "76fffeedcdc34a34bd695005c86e18e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76fffeedcdc34a34bd695005c86e18e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 } health_report { overall_health: UNKNOWN } } }
I20260620 14:29:41.934630 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.931353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.0.98.3:42291)
0620 14:29:41.931490 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:41.931494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:41.931509 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:29:41.931581 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:41.931583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:41.931632 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:41.931716 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:29:41.932006 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.932478 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.933342 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:41.933542 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:41.933615 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:41.933856 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:29:41.934270 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:29:41.934358 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:41.934413 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:29:41.934455 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:29:41.940330 20282 raft_consensus.cc:1261] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5 [term 1 FOLLOWER]: Refusing update from remote peer 76fffeedcdc34a34bd695005c86e18e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:41.940402 20413 raft_consensus.cc:1261] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe [term 1 FOLLOWER]: Refusing update from remote peer 76fffeedcdc34a34bd695005c86e18e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:29:41.941215 20615 consensus_queue.cc:1035] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5480d22306924b3e9ac7aaf5edf9d6fe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:41.941589 20617 consensus_queue.cc:1035] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63284f059b8f4622bfce7c4d52a64df5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:29:41.950851 20620 mvcc.cc:205] Tried to move back new op lower bound from 7298931842820071424 to 7298931842550038528. Current Snapshot: MvccSnapshot[applied={T|T < 7298931842820071424}]
I20260620 14:29:41.952505 20622 mvcc.cc:205] Tried to move back new op lower bound from 7298931842820071424 to 7298931842550038528. Current Snapshot: MvccSnapshot[applied={T|T < 7298931842820071424}]
I20260620 14:29:41.953438 20621 mvcc.cc:205] Tried to move back new op lower bound from 7298931842820071424 to 7298931842550038528. Current Snapshot: MvccSnapshot[applied={T|T < 7298931842820071424}]
I20260620 14:29:41.959780 20140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:41.959913 20140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:29:41.962080 20140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d559cae7d3a6474a8504e268d7e02fa3: Sending DeleteTablet for 3 replicas of tablet 067d10fde3c54269a83d892f85f14297
I20260620 14:29:41.962785 20262 tablet_service.cc:1514] Processing DeleteTablet for tablet 067d10fde3c54269a83d892f85f14297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:41 UTC) from {username='slave'} at 127.0.0.1:52958
I20260620 14:29:41.962826 20524 tablet_service.cc:1514] Processing DeleteTablet for tablet 067d10fde3c54269a83d892f85f14297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:41 UTC) from {username='slave'} at 127.0.0.1:44458
I20260620 14:29:41.963222 20629 tablet_replica.cc:330] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5: stopping tablet replica
I20260620 14:29:41.963313 20630 tablet_replica.cc:330] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: stopping tablet replica
I20260620 14:29:41.963409 20393 tablet_service.cc:1514] Processing DeleteTablet for tablet 067d10fde3c54269a83d892f85f14297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:29:41 UTC) from {username='slave'} at 127.0.0.1:59822
I20260620 14:29:41.963577 20630 raft_consensus.cc:2229] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:29:41.963689 20629 raft_consensus.cc:2229] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:41.963948 20630 raft_consensus.cc:2258] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:41.964118 20629 raft_consensus.cc:2258] T 067d10fde3c54269a83d892f85f14297 P 63284f059b8f4622bfce7c4d52a64df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:41.964527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 20200
I20260620 14:29:41.965387 20630 ts_tablet_manager.cc:1905] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:41.967316 20631 tablet_replica.cc:330] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: stopping tablet replica
I20260620 14:29:41.967657 20631 raft_consensus.cc:2229] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:29:41.968007 20631 raft_consensus.cc:2258] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:29:41.970310 20631 ts_tablet_manager.cc:1905] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:29:41.972400 20631 ts_tablet_manager.cc:1918] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:41.972507 20631 log.cc:1196] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-1/wal/wals/067d10fde3c54269a83d892f85f14297
I20260620 14:29:41.972846 20631 ts_tablet_manager.cc:1939] T 067d10fde3c54269a83d892f85f14297 P 5480d22306924b3e9ac7aaf5edf9d6fe: Deleting consensus metadata
W20260620 14:29:41.972993 20126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32815 (error 108)
I20260620 14:29:41.973733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 20331
I20260620 14:29:41.975260 20630 ts_tablet_manager.cc:1918] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:29:41.975538 20630 log.cc:1196] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: Deleting WAL directory at /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781965536974027-392-0/minicluster-data/ts-2/wal/wals/067d10fde3c54269a83d892f85f14297
I20260620 14:29:41.975910 20630 ts_tablet_manager.cc:1939] T 067d10fde3c54269a83d892f85f14297 P 76fffeedcdc34a34bd695005c86e18e2: Deleting consensus metadata
I20260620 14:29:41.976867 20124 catalog_manager.cc:4841] TS 76fffeedcdc34a34bd695005c86e18e2 (127.0.98.3:42291): tablet 067d10fde3c54269a83d892f85f14297 (table test-table [id=870a8fa285fa411c81b28864f67dd990]) successfully deleted
I20260620 14:29:41.977691 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.977327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32815 (local address 127.0.0.1:52968)
0620 14:29:41.977517 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:41.977592 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 14:29:41.977970 20126 catalog_manager.cc:4584] TS 63284f059b8f4622bfce7c4d52a64df5 (127.0.98.1:32815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 067d10fde3c54269a83d892f85f14297: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32815: connect: Connection refused (error 111)
I20260620 14:29:41.981490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 20462
I20260620 14:29:41.982632 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:41.982308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45599 (local address 127.0.0.1:59834)
0620 14:29:41.982489 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:29:41.982547 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 14:29:41.987059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 20109
2026-06-20T14:29:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1252 ms)
[----------] 6 tests from EncryptionPolicyTest (10519 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:42 dist-test-slave-pk3j krb5kdc[20635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:42 dist-test-slave-pk3j krb5kdc[20635](info): set up 2 sockets
Jun 20 14:29:42 dist-test-slave-pk3j krb5kdc[20635](info): commencing operation
krb5kdc: starting...
W20260620 14:29:43.013186 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:43 dist-test-slave-pk3j krb5kdc[20635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:43.123629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:43.229175 20648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:43.229501 20648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:43.229579 20648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:43.233500 20648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:43.233570 20648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:43.233605 20648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:43.233656 20648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:43.233690 20648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:43.238834 20648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:43.239796 20648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:43.240741 20648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:43.248562 20656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:43.248637 20653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:43.248624 20648 server_base.cc:1034] running on GCE node
W20260620 14:29:43.248580 20654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:43.249320 20648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:43.250274 20648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:43.251458 20648 hybrid_clock.cc:648] HybridClock initialized: now 1781965783251451 us; error 28 us; skew 500 ppm
Jun 20 14:29:43 dist-test-slave-pk3j krb5kdc[20635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:43.254073 20648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:43.254818 20648 webserver.cc:460] Webserver started at http://127.0.0.1:33357/ using document root <none> and password file <none>
I20260620 14:29:43.255281 20648 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:43.255335 20648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:43.255489 20648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:43.257382 20648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "808cf8f2e1f14cab889a679f2d1e766d"
format_stamp: "Formatted at 2026-06-20 14:29:43 on dist-test-slave-pk3j"
server_key: "26c1a47a09663843f03c6ad63cbbdfdf"
server_key_iv: "e1aef1204c33e19f278053822b8bb6de"
server_key_version: "encryptionkey@0"
I20260620 14:29:43.257911 20648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "808cf8f2e1f14cab889a679f2d1e766d"
format_stamp: "Formatted at 2026-06-20 14:29:43 on dist-test-slave-pk3j"
server_key: "26c1a47a09663843f03c6ad63cbbdfdf"
server_key_iv: "e1aef1204c33e19f278053822b8bb6de"
server_key_version: "encryptionkey@0"
I20260620 14:29:43.261370 20648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:29:43.263521 20663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:43.264637 20648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:43.264729 20648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "808cf8f2e1f14cab889a679f2d1e766d"
format_stamp: "Formatted at 2026-06-20 14:29:43 on dist-test-slave-pk3j"
server_key: "26c1a47a09663843f03c6ad63cbbdfdf"
server_key_iv: "e1aef1204c33e19f278053822b8bb6de"
server_key_version: "encryptionkey@0"
I20260620 14:29:43.264812 20648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:43.284242 20648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:43.286836 20648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:43.286966 20648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:43.293458 20648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43201
I20260620 14:29:43.293499 20715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43201 every 8 connection(s)
I20260620 14:29:43.294330 20648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:43.296787 20716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:43.298578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 20648
I20260620 14:29:43.298717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:43.298998 392 external_mini_cluster.cc:1428] Setting key 0ceb8e50234c1269da1640fc1691f5f5
I20260620 14:29:43.302292 20716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d: Bootstrap starting.
I20260620 14:29:43.304335 20716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:43.305038 20716 log.cc:824] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:43.306874 20716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d: No bootstrap required, opened a new log
Jun 20 14:29:43 dist-test-slave-pk3j krb5kdc[20635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:29:43.309696 20716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808cf8f2e1f14cab889a679f2d1e766d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260620 14:29:43.309921 20716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:43.310000 20716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808cf8f2e1f14cab889a679f2d1e766d, State: Initialized, Role: FOLLOWER
I20260620 14:29:43.310434 20716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808cf8f2e1f14cab889a679f2d1e766d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260620 14:29:43.310577 20716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:43.310645 20716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:43.310789 20716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:43.311568 20716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808cf8f2e1f14cab889a679f2d1e766d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260620 14:29:43.311887 20716 leader_election.cc:302] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 808cf8f2e1f14cab889a679f2d1e766d; no voters:
I20260620 14:29:43.312211 20716 leader_election.cc:288] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:43.312259 20721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:43.312445 20721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [term 1 LEADER]: Becoming Leader. State: Replica: 808cf8f2e1f14cab889a679f2d1e766d, State: Running, Role: LEADER
I20260620 14:29:43.312713 20721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808cf8f2e1f14cab889a679f2d1e766d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } }
I20260620 14:29:43.313098 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:43.300292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56857 (local address 127.0.0.1:43201)
0620 14:29:43.300671 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:43.300685 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:43.300717 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:29:43.301493 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:43.301510 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:43.301829 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:43.302141 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:43.302882 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:43.303811 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:43.304723 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:43.305207 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:43.305406 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:43.307872 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:43.307880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:43.307897 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:43.307921 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:43.310156 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:43.310625 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:43.310630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:43.310635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:43.310819 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:43.311180 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:43.311183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:43.311185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:43.311526 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:43.311744 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:43.312001 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 14:29:43.312206 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260620 14:29:43.313431 20716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:43.313796 20721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "808cf8f2e1f14cab889a679f2d1e766d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808cf8f2e1f14cab889a679f2d1e766d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } } }
I20260620 14:29:43.313900 20721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:43.314517 20722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 808cf8f2e1f14cab889a679f2d1e766d. Latest consensus state: current_term: 1 leader_uuid: "808cf8f2e1f14cab889a679f2d1e766d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "808cf8f2e1f14cab889a679f2d1e766d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43201 } } }
I20260620 14:29:43.314612 20722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:43.314857 20725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:43.316092 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:43.317845 20725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:43.326248 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:43.318328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40625 (local address 127.0.0.1:43201)
0620 14:29:43.318432 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:43.318436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:43.318718 (+ 282us) server_negotiation.cc:407] Connection header received
0620 14:29:43.318865 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:43.318869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:43.318913 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:43.319002 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:43.319598 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:43.320328 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:43.321477 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:43.321765 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:43.321909 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:43.323209 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:43.323212 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:43.323214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:43.323237 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:43.324522 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:43.324980 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:43.324984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:43.324985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:43.325063 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:43.325435 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:43.325439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:43.325440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:43.325594 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:43.325738 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:43.325995 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 14:29:43.326080 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:29:43.326663 20725 catalog_manager.cc:1347] Generated new cluster ID: 2eb17bb8f91a4551ac5188554fcada7d
I20260620 14:29:43.326717 20725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:43.347091 20725 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:43.347946 20725 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:43.360807 20725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 808cf8f2e1f14cab889a679f2d1e766d: Generated new TSK 0
I20260620 14:29:43.361743 20725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:43.375586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 20648
W20260620 14:29:43.381034 20633 connection.cc:535] client connection to 127.0.0.1:43201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43201): Connection reset by peer (error 104)
2026-06-20T14:29:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:43 dist-test-slave-pk3j krb5kdc[20747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:43 dist-test-slave-pk3j krb5kdc[20747](info): set up 2 sockets
Jun 20 14:29:43 dist-test-slave-pk3j krb5kdc[20747](info): commencing operation
krb5kdc: starting...
W20260620 14:29:44.381186 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:44 dist-test-slave-pk3j krb5kdc[20747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:44.491986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:44.596968 20760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:44.597236 20760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:44.597313 20760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:44.601046 20760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:44.601125 20760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:44.601163 20760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:44.601212 20760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:44.601248 20760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:44.606093 20760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:44.606973 20760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:44.607863 20760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:44.615619 20766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:44.615640 20768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:44.615640 20765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:44.615801 20760 server_base.cc:1034] running on GCE node
I20260620 14:29:44.616508 20760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:44.617564 20760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:44.618726 20760 hybrid_clock.cc:648] HybridClock initialized: now 1781965784618709 us; error 30 us; skew 500 ppm
Jun 20 14:29:44 dist-test-slave-pk3j krb5kdc[20747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:44.621367 20760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:44.622157 20760 webserver.cc:460] Webserver started at http://127.0.0.1:33813/ using document root <none> and password file <none>
I20260620 14:29:44.622639 20760 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:44.622694 20760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:44.622843 20760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:44.624627 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "ea9bad6940db463f89bcc905d793796c"
format_stamp: "Formatted at 2026-06-20 14:29:44 on dist-test-slave-pk3j"
server_key: "442baf3a59021d4aa9f0bcfe80f17d99"
server_key_iv: "10746ad5ddf6197198973342173482ba"
server_key_version: "encryptionkey@0"
I20260620 14:29:44.625103 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea9bad6940db463f89bcc905d793796c"
format_stamp: "Formatted at 2026-06-20 14:29:44 on dist-test-slave-pk3j"
server_key: "442baf3a59021d4aa9f0bcfe80f17d99"
server_key_iv: "10746ad5ddf6197198973342173482ba"
server_key_version: "encryptionkey@0"
I20260620 14:29:44.628418 20760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:29:44.630568 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:44.631701 20760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:29:44.631794 20760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "ea9bad6940db463f89bcc905d793796c"
format_stamp: "Formatted at 2026-06-20 14:29:44 on dist-test-slave-pk3j"
server_key: "442baf3a59021d4aa9f0bcfe80f17d99"
server_key_iv: "10746ad5ddf6197198973342173482ba"
server_key_version: "encryptionkey@0"
I20260620 14:29:44.631871 20760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:44.643600 20760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:44.646087 20760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:44.646221 20760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:44.652763 20760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37677
I20260620 14:29:44.652779 20827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37677 every 8 connection(s)
I20260620 14:29:44.653632 20760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:44.656175 20828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:44.657153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 20760
I20260620 14:29:44.657321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:44.657652 392 external_mini_cluster.cc:1428] Setting key 6e0185107328376083da96d4aadb57b3
I20260620 14:29:44.662232 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c: Bootstrap starting.
I20260620 14:29:44.664673 20828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:44.665518 20828 log.cc:824] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:44.666280 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:44.658866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47083 (local address 127.0.0.1:37677)
0620 14:29:44.659273 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:44.659291 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:44.659325 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:29:44.659922 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:44.659938 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:44.660257 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:44.660507 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:44.660825 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:44.661811 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:44.662958 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:44.663381 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:44.663491 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:44.664309 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:44.664314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:44.664324 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:44.664688 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:44.664788 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:44.664829 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:44.664928 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:44.665178 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 14:29:44.665349 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":141,"threads_started":1}
I20260620 14:29:44.667452 20828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c: No bootstrap required, opened a new log
I20260620 14:29:44.669080 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:44.671860 20828 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bad6940db463f89bcc905d793796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } }
I20260620 14:29:44.672134 20828 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:44.672194 20828 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9bad6940db463f89bcc905d793796c, State: Initialized, Role: FOLLOWER
I20260620 14:29:44.672567 20828 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bad6940db463f89bcc905d793796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } }
I20260620 14:29:44.672677 20828 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:44.672768 20828 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:44.672860 20828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:44.673887 20828 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bad6940db463f89bcc905d793796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } }
I20260620 14:29:44.674280 20828 leader_election.cc:302] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9bad6940db463f89bcc905d793796c; no voters:
I20260620 14:29:44.674604 20828 leader_election.cc:288] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:44.674707 20838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:44.674943 20838 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [term 1 LEADER]: Becoming Leader. State: Replica: ea9bad6940db463f89bcc905d793796c, State: Running, Role: LEADER
I20260620 14:29:44.675248 20838 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bad6940db463f89bcc905d793796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } }
I20260620 14:29:44.675726 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:44.671376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43113 (local address 127.0.0.1:37677)
0620 14:29:44.671480 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:44.671484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:44.671609 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:29:44.671725 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:44.671727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:44.671759 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:44.671824 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:44.672145 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:44.672801 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:44.673925 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:44.674189 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:44.674270 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:44.675224 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:44.675228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:44.675230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:44.675243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:44.675268 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:44.675281 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:44.675340 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:44.675588 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:29:44.675618 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:29:44.675904 20828 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:44.676322 20838 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9bad6940db463f89bcc905d793796c. Latest consensus state: current_term: 1 leader_uuid: "ea9bad6940db463f89bcc905d793796c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bad6940db463f89bcc905d793796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } } }
I20260620 14:29:44.676505 20838 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:44.676705 20839 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9bad6940db463f89bcc905d793796c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9bad6940db463f89bcc905d793796c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37677 } } }
I20260620 14:29:44.676862 20839 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:44.676998 20845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:44.679658 20845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:44.684194 20845 catalog_manager.cc:1347] Generated new cluster ID: c8a982fa94ff4f3da8f2f573af1f9215
I20260620 14:29:44.684257 20845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:44.698809 20845 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:44.699708 20845 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:44.705897 20845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9bad6940db463f89bcc905d793796c: Generated new TSK 0
I20260620 14:29:44.706593 20845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:44.720531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 20760
W20260620 14:29:44.725998 20745 connection.cc:535] client connection to 127.0.0.1:37677 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37677): Connection reset by peer (error 104)
2026-06-20T14:29:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:44 dist-test-slave-pk3j krb5kdc[20859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:44 dist-test-slave-pk3j krb5kdc[20859](info): set up 2 sockets
Jun 20 14:29:44 dist-test-slave-pk3j krb5kdc[20859](info): commencing operation
krb5kdc: starting...
W20260620 14:29:45.753180 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:45 dist-test-slave-pk3j krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:45.861918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:45.967051 20872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:45.967307 20872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:45.967412 20872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:45.971432 20872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:45.971517 20872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:45.971555 20872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:45.971612 20872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:45.971652 20872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:45.976866 20872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:45.977882 20872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:45.978840 20872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:45.989516 20877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:45.989883 20880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:45.989890 20872 server_base.cc:1034] running on GCE node
W20260620 14:29:45.991389 20878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:45.991961 20872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:45.993080 20872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:45.994272 20872 hybrid_clock.cc:648] HybridClock initialized: now 1781965785994257 us; error 25 us; skew 500 ppm
Jun 20 14:29:45 dist-test-slave-pk3j krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:45.996879 20872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:45.997829 20872 webserver.cc:460] Webserver started at http://127.0.0.1:42921/ using document root <none> and password file <none>
I20260620 14:29:45.998368 20872 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:45.998454 20872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:45.998673 20872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:46.000598 20872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "c3a4d4fdb70647a4b1a2fd286c64e756"
format_stamp: "Formatted at 2026-06-20 14:29:45 on dist-test-slave-pk3j"
server_key: "be30d71394fbb5683007b6dbe186cd20"
server_key_iv: "8f0e4a857a3b5e454f3a28bb8dd49820"
server_key_version: "encryptionkey@0"
I20260620 14:29:46.001188 20872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3a4d4fdb70647a4b1a2fd286c64e756"
format_stamp: "Formatted at 2026-06-20 14:29:45 on dist-test-slave-pk3j"
server_key: "be30d71394fbb5683007b6dbe186cd20"
server_key_iv: "8f0e4a857a3b5e454f3a28bb8dd49820"
server_key_version: "encryptionkey@0"
I20260620 14:29:46.004722 20872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:29:46.007082 20887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:46.008252 20872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:46.008410 20872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "c3a4d4fdb70647a4b1a2fd286c64e756"
format_stamp: "Formatted at 2026-06-20 14:29:45 on dist-test-slave-pk3j"
server_key: "be30d71394fbb5683007b6dbe186cd20"
server_key_iv: "8f0e4a857a3b5e454f3a28bb8dd49820"
server_key_version: "encryptionkey@0"
I20260620 14:29:46.008524 20872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:46.012928 20872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:46.015734 20872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:46.015928 20872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:46.023058 20872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42777
I20260620 14:29:46.023068 20939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42777 every 8 connection(s)
I20260620 14:29:46.023963 20872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:46.026561 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:46.026883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 20872
I20260620 14:29:46.027035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:46.027351 392 external_mini_cluster.cc:1428] Setting key 941afd39bed19f421a2d9cf1cbace70a
I20260620 14:29:46.032344 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:46.028701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38091 (local address 127.0.0.1:42777)
0620 14:29:46.029116 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:46.029130 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:46.029165 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:29:46.029879 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:46.029895 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:46.030151 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:46.030466 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:46.031264 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:46.031268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:46.031279 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:46.031289 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:46.031340 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:46.031358 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:46.031502 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:46.031570 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:29:46.031723 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":128,"threads_started":1}
I20260620 14:29:46.033105 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756: Bootstrap starting.
I20260620 14:29:46.035106 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:46.035636 20940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:46.036386 20940 log.cc:824] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:46.038681 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756: No bootstrap required, opened a new log
I20260620 14:29:46.038729 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:46.037181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58953 (local address 127.0.0.1:42777)
0620 14:29:46.037294 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:46.037298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:46.037527 (+ 229us) server_negotiation.cc:407] Connection header received
0620 14:29:46.037637 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:46.037639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:46.037666 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:46.037722 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:46.038345 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:46.038347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:46.038349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:46.038358 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:46.038379 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:46.038389 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:46.038453 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:46.038564 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 14:29:46.038592 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:29:46.042083 20940 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a4d4fdb70647a4b1a2fd286c64e756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260620 14:29:46.042320 20940 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:46.042394 20940 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a4d4fdb70647a4b1a2fd286c64e756, State: Initialized, Role: FOLLOWER
I20260620 14:29:46.042860 20940 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a4d4fdb70647a4b1a2fd286c64e756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260620 14:29:46.042996 20950 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:29:46.043154 20950 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a4d4fdb70647a4b1a2fd286c64e756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260620 14:29:46.043489 20940 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:46.043483 20950 leader_election.cc:302] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3a4d4fdb70647a4b1a2fd286c64e756; no voters:
I20260620 14:29:46.043582 20940 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:46.043649 20940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:46.043841 20950 leader_election.cc:288] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260620 14:29:46.044564 20940 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a4d4fdb70647a4b1a2fd286c64e756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260620 14:29:46.044703 20940 leader_election.cc:302] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3a4d4fdb70647a4b1a2fd286c64e756; no voters:
I20260620 14:29:46.044780 20951 raft_consensus.cc:2741] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 14:29:46.044833 20940 leader_election.cc:288] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:46.044999 20951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:46.045073 20951 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [term 1 LEADER]: Becoming Leader. State: Replica: c3a4d4fdb70647a4b1a2fd286c64e756, State: Running, Role: LEADER
I20260620 14:29:46.045356 20951 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a4d4fdb70647a4b1a2fd286c64e756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260620 14:29:46.045979 20940 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:46.046535 20951 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3a4d4fdb70647a4b1a2fd286c64e756. Latest consensus state: current_term: 1 leader_uuid: "c3a4d4fdb70647a4b1a2fd286c64e756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a4d4fdb70647a4b1a2fd286c64e756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } } }
I20260620 14:29:46.046726 20951 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:46.046926 20952 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3a4d4fdb70647a4b1a2fd286c64e756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a4d4fdb70647a4b1a2fd286c64e756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } } }
I20260620 14:29:46.047029 20952 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:46.047152 20958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:46.050391 20958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:46.055938 20958 catalog_manager.cc:1347] Generated new cluster ID: 12aa6f3c59a44ad096aeb043006b655a
I20260620 14:29:46.056010 20958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:46.069128 20958 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:46.069433 20958 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:46.079033 20958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3a4d4fdb70647a4b1a2fd286c64e756: Generated new TSK 0
I20260620 14:29:46.079641 20958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:46.091647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 20872
2026-06-20T14:29:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:46 dist-test-slave-pk3j krb5kdc[20971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:46 dist-test-slave-pk3j krb5kdc[20971](info): set up 2 sockets
Jun 20 14:29:46 dist-test-slave-pk3j krb5kdc[20971](info): commencing operation
krb5kdc: starting...
W20260620 14:29:47.109418 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:47 dist-test-slave-pk3j krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:47.217722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:47.323119 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:47.323398 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:47.323470 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:47.327518 20984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:47.327596 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:47.327634 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:47.327690 20984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:47.327723 20984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:47.332736 20984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:47.333679 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:47.334641 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:47.342645 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:47.342653 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:47.342720 20984 server_base.cc:1034] running on GCE node
W20260620 14:29:47.342657 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:47.343397 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:47.344311 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:47.345463 20984 hybrid_clock.cc:648] HybridClock initialized: now 1781965787345435 us; error 37 us; skew 500 ppm
Jun 20 14:29:47 dist-test-slave-pk3j krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:47.354431 20984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:47.355284 20984 webserver.cc:460] Webserver started at http://127.0.0.1:42045/ using document root <none> and password file <none>
I20260620 14:29:47.355782 20984 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:47.355849 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:47.356009 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:47.357964 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "5d8c9b37c87e498f90474d7473b1d0f9"
format_stamp: "Formatted at 2026-06-20 14:29:47 on dist-test-slave-pk3j"
server_key: "7598be6ed6c7b828c96b171923ceaf4b"
server_key_iv: "24015f0722672e55abd239c2527354a5"
server_key_version: "encryptionkey@0"
I20260620 14:29:47.358470 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d8c9b37c87e498f90474d7473b1d0f9"
format_stamp: "Formatted at 2026-06-20 14:29:47 on dist-test-slave-pk3j"
server_key: "7598be6ed6c7b828c96b171923ceaf4b"
server_key_iv: "24015f0722672e55abd239c2527354a5"
server_key_version: "encryptionkey@0"
I20260620 14:29:47.361800 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:29:47.363977 20999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:47.365110 20984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:47.365280 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "5d8c9b37c87e498f90474d7473b1d0f9"
format_stamp: "Formatted at 2026-06-20 14:29:47 on dist-test-slave-pk3j"
server_key: "7598be6ed6c7b828c96b171923ceaf4b"
server_key_iv: "24015f0722672e55abd239c2527354a5"
server_key_version: "encryptionkey@0"
I20260620 14:29:47.365397 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:47.374717 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:47.377691 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:47.377871 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:47.384593 20984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37403
I20260620 14:29:47.384610 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37403 every 8 connection(s)
I20260620 14:29:47.385494 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:47.387954 21052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:47.392853 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9: Bootstrap starting.
I20260620 14:29:47.393417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 20984
I20260620 14:29:47.393556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:47.393843 392 external_mini_cluster.cc:1428] Setting key 5fb29444fced9202e3413d3309e48561
I20260620 14:29:47.395558 21052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:47.396258 21052 log.cc:824] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:47.398204 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9: No bootstrap required, opened a new log
Jun 20 14:29:47 dist-test-slave-pk3j krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:29:47.402451 21052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c9b37c87e498f90474d7473b1d0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } }
I20260620 14:29:47.402685 21052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:47.402767 21052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8c9b37c87e498f90474d7473b1d0f9, State: Initialized, Role: FOLLOWER
I20260620 14:29:47.403215 21052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c9b37c87e498f90474d7473b1d0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } }
I20260620 14:29:47.403371 21052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:47.403455 21052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:47.403553 21052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:47.404655 21052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c9b37c87e498f90474d7473b1d0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } }
I20260620 14:29:47.405020 21052 leader_election.cc:302] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d8c9b37c87e498f90474d7473b1d0f9; no voters:
I20260620 14:29:47.405386 21052 leader_election.cc:288] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:47.405443 21057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:47.405622 21057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [term 1 LEADER]: Becoming Leader. State: Replica: 5d8c9b37c87e498f90474d7473b1d0f9, State: Running, Role: LEADER
I20260620 14:29:47.405953 21057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c9b37c87e498f90474d7473b1d0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } }
I20260620 14:29:47.406562 21052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:47.407615 21058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d8c9b37c87e498f90474d7473b1d0f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c9b37c87e498f90474d7473b1d0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } } }
I20260620 14:29:47.407781 21058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:47.407497 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d8c9b37c87e498f90474d7473b1d0f9. Latest consensus state: current_term: 1 leader_uuid: "5d8c9b37c87e498f90474d7473b1d0f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d8c9b37c87e498f90474d7473b1d0f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } } }
I20260620 14:29:47.408249 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:47.395083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52545 (local address 127.0.0.1:37403)
0620 14:29:47.395456 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:47.395472 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:47.395504 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:29:47.396147 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:47.396163 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:47.396519 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:47.396875 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:47.397663 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:47.398632 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:47.399740 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:47.400169 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:47.400282 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:47.403019 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:47.403024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:47.403035 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:47.403059 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:47.405038 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:47.405600 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:47.405608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:47.405616 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:47.405758 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:47.406080 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:47.406083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:47.406084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:47.406468 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:47.406669 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:47.406937 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:29:47.407238 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":126,"threads_started":1}
I20260620 14:29:47.408281 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:47.410898 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 14:29:47.411288 21072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:29:47.411365 21072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:29:47.411422 21066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:47.412282 21066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:47.417661 21066 catalog_manager.cc:1347] Generated new cluster ID: ec4799cc474b4de9bf2ef6662823ee50
I20260620 14:29:47.417714 21066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:47.421025 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:47.412907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44617 (local address 127.0.0.1:37403)
0620 14:29:47.413017 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:47.413021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:47.413263 (+ 242us) server_negotiation.cc:407] Connection header received
0620 14:29:47.413375 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:47.413378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:47.413415 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:47.413504 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:47.414113 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:47.414895 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:47.415948 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:47.416278 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:47.416380 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:47.417962 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:47.417966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:47.417969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:47.417992 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:47.419552 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:47.419963 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:47.419966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:47.419968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:47.420034 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:47.420264 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:47.420266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:47.420268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:47.420430 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:47.420510 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:47.420787 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:29:47.420893 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:29:47.425307 21066 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:47.426394 21066 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:47.438650 21066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d8c9b37c87e498f90474d7473b1d0f9: Generated new TSK 0
I20260620 14:29:47.439352 21066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:47.444095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 20984
2026-06-20T14:29:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:47 dist-test-slave-pk3j krb5kdc[21083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:47 dist-test-slave-pk3j krb5kdc[21083](info): set up 2 sockets
Jun 20 14:29:47 dist-test-slave-pk3j krb5kdc[21083](info): commencing operation
krb5kdc: starting...
W20260620 14:29:48.461220 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:48 dist-test-slave-pk3j krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:48.569723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:48.674228 21096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:48.674444 21096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:48.674482 21096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:48.678211 21096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:48.678251 21096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:48.678270 21096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:48.678285 21096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:48.678300 21096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:48.682983 21096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:48.683781 21096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:48.684677 21096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:48.692552 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:48.692576 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:48.692576 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:48.693485 21096 server_base.cc:1034] running on GCE node
I20260620 14:29:48.694042 21096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:48.695068 21096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:48.696223 21096 hybrid_clock.cc:648] HybridClock initialized: now 1781965788696212 us; error 24 us; skew 500 ppm
Jun 20 14:29:48 dist-test-slave-pk3j krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:48.698729 21096 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:48.699509 21096 webserver.cc:460] Webserver started at http://127.0.0.1:37885/ using document root <none> and password file <none>
I20260620 14:29:48.700027 21096 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:48.700111 21096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:48.700328 21096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:48.702272 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "c33e3c76494543cba7e5d1dcbbe296ab"
format_stamp: "Formatted at 2026-06-20 14:29:48 on dist-test-slave-pk3j"
server_key: "6b60f325efaf1690e89ed015feade446"
server_key_iv: "ec397664dbd4a407ebc75d79094cec72"
server_key_version: "encryptionkey@0"
I20260620 14:29:48.702814 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "c33e3c76494543cba7e5d1dcbbe296ab"
format_stamp: "Formatted at 2026-06-20 14:29:48 on dist-test-slave-pk3j"
server_key: "6b60f325efaf1690e89ed015feade446"
server_key_iv: "ec397664dbd4a407ebc75d79094cec72"
server_key_version: "encryptionkey@0"
I20260620 14:29:48.706287 21096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:29:48.708461 21111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:48.709619 21096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:48.709754 21096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "c33e3c76494543cba7e5d1dcbbe296ab"
format_stamp: "Formatted at 2026-06-20 14:29:48 on dist-test-slave-pk3j"
server_key: "6b60f325efaf1690e89ed015feade446"
server_key_iv: "ec397664dbd4a407ebc75d79094cec72"
server_key_version: "encryptionkey@0"
I20260620 14:29:48.709857 21096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:48.721058 21096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:48.723624 21096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:48.723817 21096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:48.730329 21096 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45749
I20260620 14:29:48.730394 21163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45749 every 8 connection(s)
I20260620 14:29:48.731231 21096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:48.733780 21164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:48.734802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 21096
I20260620 14:29:48.734942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:48.735271 392 external_mini_cluster.cc:1428] Setting key 414ad90fc5853cbac2b4fa3fd487ce6c
I20260620 14:29:48.739599 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab: Bootstrap starting.
I20260620 14:29:48.742262 21164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:48.743031 21164 log.cc:824] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:48.743887 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:48.736588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34595 (local address 127.0.0.1:45749)
0620 14:29:48.737002 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:48.737016 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:48.737047 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:29:48.737685 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:48.737701 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:48.738013 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:48.738329 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:48.738729 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:48.739645 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:48.740935 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:48.741428 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:48.741540 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:48.742389 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:48.742399 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:48.742410 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:48.742757 (+ 347us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:48.742805 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:48.742825 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:48.742899 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:48.743059 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:29:48.743256 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":132,"threads_started":1}
I20260620 14:29:48.744920 21164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab: No bootstrap required, opened a new log
I20260620 14:29:48.746644 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:48.748927 21164 raft_consensus.cc:348] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33e3c76494543cba7e5d1dcbbe296ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } }
I20260620 14:29:48.749235 21164 raft_consensus.cc:374] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:48.749310 21164 raft_consensus.cc:729] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33e3c76494543cba7e5d1dcbbe296ab, State: Initialized, Role: FOLLOWER
I20260620 14:29:48.749740 21164 consensus_queue.cc:260] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33e3c76494543cba7e5d1dcbbe296ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } }
I20260620 14:29:48.749883 21164 raft_consensus.cc:388] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:48.749950 21164 raft_consensus.cc:482] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:48.750041 21164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:48.750933 21164 raft_consensus.cc:504] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33e3c76494543cba7e5d1dcbbe296ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } }
I20260620 14:29:48.751276 21164 leader_election.cc:302] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c33e3c76494543cba7e5d1dcbbe296ab; no voters:
I20260620 14:29:48.751628 21164 leader_election.cc:288] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:48.751730 21174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:48.751955 21174 raft_consensus.cc:686] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [term 1 LEADER]: Becoming Leader. State: Replica: c33e3c76494543cba7e5d1dcbbe296ab, State: Running, Role: LEADER
I20260620 14:29:48.752252 21174 consensus_queue.cc:237] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33e3c76494543cba7e5d1dcbbe296ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } }
I20260620 14:29:48.752439 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:48.748490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43129 (local address 127.0.0.1:45749)
0620 14:29:48.748595 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:48.748601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:48.748746 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:29:48.748857 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:48.748859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:48.748893 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:48.748954 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:48.749268 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:48.749793 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:48.750692 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:48.750936 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:48.751047 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:48.751892 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:48.751895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:48.751898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:48.751910 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:48.751936 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:48.751949 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:48.752020 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:48.752281 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:29:48.752327 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:29:48.752962 21164 sys_catalog.cc:564] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:48.753631 21176 sys_catalog.cc:455] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c33e3c76494543cba7e5d1dcbbe296ab. Latest consensus state: current_term: 1 leader_uuid: "c33e3c76494543cba7e5d1dcbbe296ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33e3c76494543cba7e5d1dcbbe296ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } } }
I20260620 14:29:48.753880 21176 sys_catalog.cc:458] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:48.754036 21175 sys_catalog.cc:455] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c33e3c76494543cba7e5d1dcbbe296ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33e3c76494543cba7e5d1dcbbe296ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45749 } } }
I20260620 14:29:48.754132 21175 sys_catalog.cc:458] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab [sys.catalog]: This master's current role is: LEADER
W20260620 14:29:48.757750 21189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:29:48.757828 21189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:29:48.757894 21183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:48.758554 21183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:48.764070 21183 catalog_manager.cc:1347] Generated new cluster ID: 08aedb56b374410ab559f371e7b06bef
I20260620 14:29:48.764130 21183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:48.769883 21183 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:48.771018 21183 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:48.776944 21183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c33e3c76494543cba7e5d1dcbbe296ab: Generated new TSK 0
I20260620 14:29:48.777616 21183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:48.789111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 21096
2026-06-20T14:29:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:48 dist-test-slave-pk3j krb5kdc[21195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:48 dist-test-slave-pk3j krb5kdc[21195](info): set up 2 sockets
Jun 20 14:29:48 dist-test-slave-pk3j krb5kdc[21195](info): commencing operation
krb5kdc: starting...
W20260620 14:29:49.797405 392 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:49 dist-test-slave-pk3j krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:49.905439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:50.017896 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:50.018138 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:50.018214 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:50.021898 21208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:50.021963 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:50.021997 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:50.022050 21208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:50.022086 21208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:50.027093 21208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:50.027949 21208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:50.028936 21208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:50.036688 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:50.036687 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:50.036769 21208 server_base.cc:1034] running on GCE node
W20260620 14:29:50.036695 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:50.037386 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:50.038308 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:50.039454 21208 hybrid_clock.cc:648] HybridClock initialized: now 1781965790039432 us; error 53 us; skew 500 ppm
Jun 20 14:29:50 dist-test-slave-pk3j krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:50.041934 21208 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:50.042689 21208 webserver.cc:460] Webserver started at http://127.0.0.1:43821/ using document root <none> and password file <none>
I20260620 14:29:50.043190 21208 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:50.043279 21208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:50.043494 21208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:50.045349 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "edbbd5516d554ad29d0237fb1803f4a5"
format_stamp: "Formatted at 2026-06-20 14:29:50 on dist-test-slave-pk3j"
server_key: "c2317ae0ba33025227cbe56158b37a60"
server_key_iv: "859126504674f65a2cbb77b08cc4a276"
server_key_version: "encryptionkey@0"
I20260620 14:29:50.045876 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "edbbd5516d554ad29d0237fb1803f4a5"
format_stamp: "Formatted at 2026-06-20 14:29:50 on dist-test-slave-pk3j"
server_key: "c2317ae0ba33025227cbe56158b37a60"
server_key_iv: "859126504674f65a2cbb77b08cc4a276"
server_key_version: "encryptionkey@0"
I20260620 14:29:50.049114 21208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:29:50.051275 21223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:50.052340 21208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:50.052474 21208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "edbbd5516d554ad29d0237fb1803f4a5"
format_stamp: "Formatted at 2026-06-20 14:29:50 on dist-test-slave-pk3j"
server_key: "c2317ae0ba33025227cbe56158b37a60"
server_key_iv: "859126504674f65a2cbb77b08cc4a276"
server_key_version: "encryptionkey@0"
I20260620 14:29:50.052577 21208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:50.056699 21208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:50.059144 21208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:50.059321 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:50.065661 21208 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34911
I20260620 14:29:50.065675 21275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34911 every 8 connection(s)
I20260620 14:29:50.066565 21208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:50.069372 21276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:50.070676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 21208
I20260620 14:29:50.070842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:50.071156 392 external_mini_cluster.cc:1428] Setting key e81b50ca901928780de1cf4b7299504a
I20260620 14:29:50.075135 21276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5: Bootstrap starting.
I20260620 14:29:50.076603 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:50.072305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51261 (local address 127.0.0.1:34911)
0620 14:29:50.072676 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:50.072693 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:50.073565 (+ 872us) server_negotiation.cc:407] Connection header received
0620 14:29:50.074241 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:50.074257 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:50.074504 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:50.074753 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:50.075522 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:50.075527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:50.075537 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:50.075548 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:50.075595 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:50.075612 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:50.075715 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:50.075830 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:29:50.075986 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":96,"threads_started":1}
I20260620 14:29:50.077387 21276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:50.078177 21276 log.cc:824] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:50.079231 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:50.079977 21276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5: No bootstrap required, opened a new log
I20260620 14:29:50.082608 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:50.081098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48933 (local address 127.0.0.1:34911)
0620 14:29:50.081232 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:50.081236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:50.081407 (+ 171us) server_negotiation.cc:407] Connection header received
0620 14:29:50.081524 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:50.081526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:50.081548 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:50.081630 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:50.082259 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:50.082261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:50.082263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:50.082273 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:50.082296 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:50.082305 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:50.082371 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:50.082474 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:29:50.082498 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:29:50.083933 21276 raft_consensus.cc:348] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbbd5516d554ad29d0237fb1803f4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } }
I20260620 14:29:50.084144 21276 raft_consensus.cc:374] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:50.084197 21276 raft_consensus.cc:729] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edbbd5516d554ad29d0237fb1803f4a5, State: Initialized, Role: FOLLOWER
I20260620 14:29:50.084547 21276 consensus_queue.cc:260] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbbd5516d554ad29d0237fb1803f4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } }
I20260620 14:29:50.084684 21276 raft_consensus.cc:388] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:50.084771 21276 raft_consensus.cc:482] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:50.084890 21276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:50.085734 21276 raft_consensus.cc:504] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbbd5516d554ad29d0237fb1803f4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } }
I20260620 14:29:50.086054 21276 leader_election.cc:302] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edbbd5516d554ad29d0237fb1803f4a5; no voters:
I20260620 14:29:50.086342 21276 leader_election.cc:288] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:50.086396 21286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:50.086660 21286 raft_consensus.cc:686] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [term 1 LEADER]: Becoming Leader. State: Replica: edbbd5516d554ad29d0237fb1803f4a5, State: Running, Role: LEADER
I20260620 14:29:50.086987 21286 consensus_queue.cc:237] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbbd5516d554ad29d0237fb1803f4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } }
I20260620 14:29:50.087363 21276 sys_catalog.cc:564] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:50.088402 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edbbd5516d554ad29d0237fb1803f4a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbbd5516d554ad29d0237fb1803f4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } } }
I20260620 14:29:50.088537 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:50.088922 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edbbd5516d554ad29d0237fb1803f4a5. Latest consensus state: current_term: 1 leader_uuid: "edbbd5516d554ad29d0237fb1803f4a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbbd5516d554ad29d0237fb1803f4a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34911 } } }
I20260620 14:29:50.089285 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:50.089447 21295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:50.091894 21295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:50.096817 21295 catalog_manager.cc:1347] Generated new cluster ID: 48e164383f1042499d51c24a3435a1ad
I20260620 14:29:50.096884 21295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:50.111505 21295 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:50.111759 21295 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:50.118278 21295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edbbd5516d554ad29d0237fb1803f4a5: Generated new TSK 0
I20260620 14:29:50.118875 21295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:50.125816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 21208
2026-06-20T14:29:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:50 dist-test-slave-pk3j krb5kdc[21307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:50 dist-test-slave-pk3j krb5kdc[21307](info): set up 2 sockets
Jun 20 14:29:50 dist-test-slave-pk3j krb5kdc[21307](info): commencing operation
krb5kdc: starting...
W20260620 14:29:51.117224 392 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:51 dist-test-slave-pk3j krb5kdc[21307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:51.224027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:51.327129 21320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:51.327337 21320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:51.327374 21320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:51.331338 21320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:51.331398 21320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:51.331418 21320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:51.331435 21320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:51.331450 21320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:51.336547 21320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:51.337354 21320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:51.338209 21320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:51.345815 21328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:51.345827 21325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:51.345872 21320 server_base.cc:1034] running on GCE node
W20260620 14:29:51.345834 21326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:51.346585 21320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:51.347529 21320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:51.348727 21320 hybrid_clock.cc:648] HybridClock initialized: now 1781965791348703 us; error 37 us; skew 500 ppm
Jun 20 14:29:51 dist-test-slave-pk3j krb5kdc[21307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:51.351423 21320 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:51.352226 21320 webserver.cc:460] Webserver started at http://127.0.0.1:40699/ using document root <none> and password file <none>
I20260620 14:29:51.352725 21320 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:51.352813 21320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:51.353020 21320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:51.354864 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "0f9d369d27a346d6bc58df2efdb3d553"
format_stamp: "Formatted at 2026-06-20 14:29:51 on dist-test-slave-pk3j"
server_key: "25a628b987e638d35a03995d117daffe"
server_key_iv: "d695300984d4a2c1f40c0da7c9d81a61"
server_key_version: "encryptionkey@0"
I20260620 14:29:51.355420 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f9d369d27a346d6bc58df2efdb3d553"
format_stamp: "Formatted at 2026-06-20 14:29:51 on dist-test-slave-pk3j"
server_key: "25a628b987e638d35a03995d117daffe"
server_key_iv: "d695300984d4a2c1f40c0da7c9d81a61"
server_key_version: "encryptionkey@0"
I20260620 14:29:51.358734 21320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:29:51.360913 21335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:51.362186 21320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:51.362317 21320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "0f9d369d27a346d6bc58df2efdb3d553"
format_stamp: "Formatted at 2026-06-20 14:29:51 on dist-test-slave-pk3j"
server_key: "25a628b987e638d35a03995d117daffe"
server_key_iv: "d695300984d4a2c1f40c0da7c9d81a61"
server_key_version: "encryptionkey@0"
I20260620 14:29:51.362429 21320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:51.372942 21320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:51.375407 21320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:51.375593 21320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:51.382133 21320 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43705
I20260620 14:29:51.382145 21387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43705 every 8 connection(s)
I20260620 14:29:51.382908 21320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:51.385372 21388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:51.389323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 21320
I20260620 14:29:51.389467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:51.389746 392 external_mini_cluster.cc:1428] Setting key 0f8c0293adcc12f97029b3773b5785d4
I20260620 14:29:51.390391 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553: Bootstrap starting.
I20260620 14:29:51.392498 21388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:51.393165 21388 log.cc:824] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:51.394922 21388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553: No bootstrap required, opened a new log
Jun 20 14:29:51 dist-test-slave-pk3j krb5kdc[21307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:29:51.398655 21388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9d369d27a346d6bc58df2efdb3d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260620 14:29:51.398844 21388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:51.398909 21388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9d369d27a346d6bc58df2efdb3d553, State: Initialized, Role: FOLLOWER
I20260620 14:29:51.399305 21388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9d369d27a346d6bc58df2efdb3d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260620 14:29:51.399420 21388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:51.399498 21388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:51.399591 21388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:51.400319 21388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9d369d27a346d6bc58df2efdb3d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260620 14:29:51.400615 21388 leader_election.cc:302] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f9d369d27a346d6bc58df2efdb3d553; no voters:
I20260620 14:29:51.400913 21388 leader_election.cc:288] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:51.400997 21393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:51.401194 21393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [term 1 LEADER]: Becoming Leader. State: Replica: 0f9d369d27a346d6bc58df2efdb3d553, State: Running, Role: LEADER
I20260620 14:29:51.401557 21393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9d369d27a346d6bc58df2efdb3d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } }
I20260620 14:29:51.402092 21388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:51.403934 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:51.390987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:45953 (local address 127.0.0.1:43705)
0620 14:29:51.391357 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:51.391369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:51.391400 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:29:51.391954 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:51.391969 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:51.392264 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:51.392560 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:51.393352 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:51.394469 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:51.395367 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:51.395818 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:51.395995 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:51.398673 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:51.398678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:51.398689 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:51.398713 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:51.400437 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:51.401107 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:51.401114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:51.401119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:51.401287 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:51.401672 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:51.401675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:51.401677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:51.402117 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:51.402324 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:51.402926 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 14:29:51.403156 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
I20260620 14:29:51.404672 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f9d369d27a346d6bc58df2efdb3d553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9d369d27a346d6bc58df2efdb3d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } } }
I20260620 14:29:51.405833 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:51.404810 21395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f9d369d27a346d6bc58df2efdb3d553. Latest consensus state: current_term: 1 leader_uuid: "0f9d369d27a346d6bc58df2efdb3d553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f9d369d27a346d6bc58df2efdb3d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43705 } } }
I20260620 14:29:51.406162 21395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553 [sys.catalog]: This master's current role is: LEADER
W20260620 14:29:51.406857 21407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 14:29:51.407123 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 14:29:51.407130 21407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:29:51.408192 21410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:51.409070 21410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:51.415143 21410 catalog_manager.cc:1347] Generated new cluster ID: 0f77ca96f11b44249f73211fbd5fa5a3
I20260620 14:29:51.415225 21410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:51.417575 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:51.408902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:46189 (local address 127.0.0.1:43705)
0620 14:29:51.409004 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:51.409008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:51.409470 (+ 462us) server_negotiation.cc:407] Connection header received
0620 14:29:51.409637 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:51.409640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:51.409679 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:51.409768 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:51.410361 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:51.410897 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:51.411969 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:51.412319 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:51.412458 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:51.414238 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:51.414241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:51.414244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:51.414268 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:51.415794 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:51.416255 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:51.416258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:51.416260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:51.416333 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:51.416673 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:51.416677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:51.416678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:51.416869 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:51.417019 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:51.417316 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 14:29:51.417416 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:29:51.436187 21410 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:51.437036 21410 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:51.445079 21410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f9d369d27a346d6bc58df2efdb3d553: Generated new TSK 0
I20260620 14:29:51.445801 21410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:51.453255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 21320
W20260620 14:29:51.458395 21305 connection.cc:535] client connection to 127.0.0.1:43705 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43705): Connection reset by peer (error 104)
2026-06-20T14:29:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:51 dist-test-slave-pk3j krb5kdc[21419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:51 dist-test-slave-pk3j krb5kdc[21419](info): set up 2 sockets
Jun 20 14:29:51 dist-test-slave-pk3j krb5kdc[21419](info): commencing operation
krb5kdc: starting...
W20260620 14:29:52.457362 392 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:52 dist-test-slave-pk3j krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:52.564687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:52.664880 21432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:52.665122 21432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:52.665160 21432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:52.668910 21432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:52.668958 21432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:52.668977 21432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:52.668992 21432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:52.669006 21432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:52.673735 21432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:52.674479 21432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:52.675289 21432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:52.682607 21437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:52.682618 21438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:52.682606 21440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:52.682842 21432 server_base.cc:1034] running on GCE node
I20260620 14:29:52.683326 21432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:52.684219 21432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:52.685369 21432 hybrid_clock.cc:648] HybridClock initialized: now 1781965792685353 us; error 29 us; skew 500 ppm
Jun 20 14:29:52 dist-test-slave-pk3j krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:52.692425 21432 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:52.693240 21432 webserver.cc:460] Webserver started at http://127.0.0.1:40199/ using document root <none> and password file <none>
I20260620 14:29:52.693717 21432 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:52.693770 21432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:52.693917 21432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:52.695632 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "5b51c0d012eb495fa41da79954e56651"
format_stamp: "Formatted at 2026-06-20 14:29:52 on dist-test-slave-pk3j"
server_key: "e3b6308fcfafc4ec289cc8a574d189e7"
server_key_iv: "31a6c5c3be49e105d693a042cad1c750"
server_key_version: "encryptionkey@0"
I20260620 14:29:52.696105 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b51c0d012eb495fa41da79954e56651"
format_stamp: "Formatted at 2026-06-20 14:29:52 on dist-test-slave-pk3j"
server_key: "e3b6308fcfafc4ec289cc8a574d189e7"
server_key_iv: "31a6c5c3be49e105d693a042cad1c750"
server_key_version: "encryptionkey@0"
I20260620 14:29:52.699326 21432 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 14:29:52.701328 21447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:52.702380 21432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:52.702469 21432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "5b51c0d012eb495fa41da79954e56651"
format_stamp: "Formatted at 2026-06-20 14:29:52 on dist-test-slave-pk3j"
server_key: "e3b6308fcfafc4ec289cc8a574d189e7"
server_key_iv: "31a6c5c3be49e105d693a042cad1c750"
server_key_version: "encryptionkey@0"
I20260620 14:29:52.702543 21432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:52.717983 21432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:52.720395 21432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:52.720525 21432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:52.726951 21432 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45403
I20260620 14:29:52.726982 21499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45403 every 8 connection(s)
I20260620 14:29:52.727794 21432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:52.729598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 21432
I20260620 14:29:52.729746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:52.730031 392 external_mini_cluster.cc:1428] Setting key c99c1aa5e585eec602b6e28f5efba3cd
I20260620 14:29:52.730556 21500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:52.736481 21500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651: Bootstrap starting.
I20260620 14:29:52.738291 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:52.731219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:46653 (local address 127.0.0.1:45403)
0620 14:29:52.731595 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:52.731618 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:52.731650 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:29:52.732235 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:52.732251 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:52.732543 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:52.732786 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:52.733091 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:52.734059 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:52.735250 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:52.735680 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:52.735786 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:52.736692 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:52.736696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:52.736708 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:52.737043 (+ 335us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:52.737094 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:52.737115 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:52.737323 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:52.737531 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 14:29:52.737674 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":135,"threads_started":1}
I20260620 14:29:52.738720 21500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:52.739351 21500 log.cc:824] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:52.740938 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:52.741168 21500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651: No bootstrap required, opened a new log
I20260620 14:29:52.744961 21500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b51c0d012eb495fa41da79954e56651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260620 14:29:52.745235 21500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:52.745296 21500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b51c0d012eb495fa41da79954e56651, State: Initialized, Role: FOLLOWER
I20260620 14:29:52.745759 21500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b51c0d012eb495fa41da79954e56651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260620 14:29:52.745934 21500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:52.746012 21500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:52.746116 21500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:52.746588 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:52.742826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:51091 (local address 127.0.0.1:45403)
0620 14:29:52.742939 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:52.742942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:52.743077 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:29:52.743194 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:52.743196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:52.743226 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:52.743286 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:52.743535 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:52.743967 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:52.744980 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:52.745195 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:52.745365 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:52.746119 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:52.746123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:52.746125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:52.746137 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:52.746160 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:52.746178 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:52.746280 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:52.746421 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 14:29:52.746448 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:29:52.747078 21500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b51c0d012eb495fa41da79954e56651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260620 14:29:52.747385 21500 leader_election.cc:302] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b51c0d012eb495fa41da79954e56651; no voters:
I20260620 14:29:52.747643 21500 leader_election.cc:288] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:52.747709 21510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:52.747929 21510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [term 1 LEADER]: Becoming Leader. State: Replica: 5b51c0d012eb495fa41da79954e56651, State: Running, Role: LEADER
I20260620 14:29:52.748232 21510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b51c0d012eb495fa41da79954e56651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } }
I20260620 14:29:52.748637 21500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:52.749801 21512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b51c0d012eb495fa41da79954e56651. Latest consensus state: current_term: 1 leader_uuid: "5b51c0d012eb495fa41da79954e56651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b51c0d012eb495fa41da79954e56651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } } }
I20260620 14:29:52.749800 21511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b51c0d012eb495fa41da79954e56651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b51c0d012eb495fa41da79954e56651" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45403 } } }
I20260620 14:29:52.749964 21512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:52.749986 21511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:52.750329 21521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:52.752919 21521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:52.757819 21521 catalog_manager.cc:1347] Generated new cluster ID: 6226b2671d634300968777297cf4d557
I20260620 14:29:52.757934 21521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:52.767310 21521 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:52.768072 21521 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:52.777418 21521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b51c0d012eb495fa41da79954e56651: Generated new TSK 0
I20260620 14:29:52.777942 21521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:52.782241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 21432
W20260620 14:29:52.787655 21417 connection.cc:535] client connection to 127.0.0.1:45403 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45403): Connection reset by peer (error 104)
2026-06-20T14:29:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:52 dist-test-slave-pk3j krb5kdc[21531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:52 dist-test-slave-pk3j krb5kdc[21531](info): set up 2 sockets
Jun 20 14:29:52 dist-test-slave-pk3j krb5kdc[21531](info): commencing operation
krb5kdc: starting...
W20260620 14:29:53.789173 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:53 dist-test-slave-pk3j krb5kdc[21531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:53.895254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:53.999590 21544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:53.999855 21544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:53.999929 21544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:54.003571 21544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:54.003645 21544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:54.003682 21544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:54.003712 21544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:54.003743 21544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:54.008562 21544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:54.009423 21544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:54.010281 21544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:54.017886 21549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:54.017922 21550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:54.018186 21544 server_base.cc:1034] running on GCE node
W20260620 14:29:54.017943 21552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:54.019019 21544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:54.020151 21544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:54.021272 21544 hybrid_clock.cc:648] HybridClock initialized: now 1781965794021254 us; error 35 us; skew 500 ppm
Jun 20 14:29:54 dist-test-slave-pk3j krb5kdc[21531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:54.023731 21544 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:54.024505 21544 webserver.cc:460] Webserver started at http://127.0.0.1:41369/ using document root <none> and password file <none>
I20260620 14:29:54.024955 21544 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:54.025008 21544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:54.025241 21544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:54.027333 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "f0da9e55527f4c899ab181e6f75b7f08"
format_stamp: "Formatted at 2026-06-20 14:29:54 on dist-test-slave-pk3j"
server_key: "74c4ad3084c02e8102a5678a6bd9e9e5"
server_key_iv: "6f7cd073fb7ba4ffb73323a87d7e7b5d"
server_key_version: "encryptionkey@0"
I20260620 14:29:54.027781 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0da9e55527f4c899ab181e6f75b7f08"
format_stamp: "Formatted at 2026-06-20 14:29:54 on dist-test-slave-pk3j"
server_key: "74c4ad3084c02e8102a5678a6bd9e9e5"
server_key_iv: "6f7cd073fb7ba4ffb73323a87d7e7b5d"
server_key_version: "encryptionkey@0"
I20260620 14:29:54.030989 21544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:29:54.033016 21559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:54.034184 21544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:54.034271 21544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "f0da9e55527f4c899ab181e6f75b7f08"
format_stamp: "Formatted at 2026-06-20 14:29:54 on dist-test-slave-pk3j"
server_key: "74c4ad3084c02e8102a5678a6bd9e9e5"
server_key_iv: "6f7cd073fb7ba4ffb73323a87d7e7b5d"
server_key_version: "encryptionkey@0"
I20260620 14:29:54.034353 21544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:54.038522 21544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:54.041034 21544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:54.041159 21544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:54.047519 21544 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37679
I20260620 14:29:54.047546 21611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37679 every 8 connection(s)
I20260620 14:29:54.048267 21544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:54.050477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 21544
I20260620 14:29:54.050612 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:54.050868 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:54.050927 392 external_mini_cluster.cc:1428] Setting key 5eee871aaeea04ab288f4da041f3c3cf
I20260620 14:29:54.055706 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:54.052198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:56973 (local address 127.0.0.1:37679)
0620 14:29:54.052571 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:54.052584 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:54.052615 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:29:54.053394 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:54.053411 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:54.053654 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:54.053949 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:54.054671 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:54.054676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:54.054686 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:54.054696 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:54.054743 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:54.054761 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:54.054902 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:54.054970 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:29:54.055109 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":122,"threads_started":1}
I20260620 14:29:54.057005 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08: Bootstrap starting.
I20260620 14:29:54.058260 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:54.059288 21612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:54.060002 21612 log.cc:824] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:54.061659 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:54.060141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:51877 (local address 127.0.0.1:37679)
0620 14:29:54.060231 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:54.060236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:54.060411 (+ 175us) server_negotiation.cc:407] Connection header received
0620 14:29:54.060570 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:54.060573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:54.060601 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:54.060685 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:54.061270 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:54.061274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:54.061276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:54.061287 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:54.061309 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:54.061320 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:54.061410 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:54.061520 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 14:29:54.061553 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:29:54.061986 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08: No bootstrap required, opened a new log
I20260620 14:29:54.065346 21612 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da9e55527f4c899ab181e6f75b7f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37679 } }
I20260620 14:29:54.065558 21612 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:54.065634 21612 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0da9e55527f4c899ab181e6f75b7f08, State: Initialized, Role: FOLLOWER
I20260620 14:29:54.066004 21612 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da9e55527f4c899ab181e6f75b7f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37679 } }
I20260620 14:29:54.066138 21612 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:54.066200 21612 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:54.066296 21612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:54.067050 21612 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da9e55527f4c899ab181e6f75b7f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37679 } }
I20260620 14:29:54.067361 21612 leader_election.cc:302] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0da9e55527f4c899ab181e6f75b7f08; no voters:
I20260620 14:29:54.067674 21612 leader_election.cc:288] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:54.067766 21622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:54.068001 21622 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [term 1 LEADER]: Becoming Leader. State: Replica: f0da9e55527f4c899ab181e6f75b7f08, State: Running, Role: LEADER
I20260620 14:29:54.068281 21622 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da9e55527f4c899ab181e6f75b7f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37679 } }
I20260620 14:29:54.068763 21612 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:54.069768 21623 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0da9e55527f4c899ab181e6f75b7f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da9e55527f4c899ab181e6f75b7f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37679 } } }
I20260620 14:29:54.069897 21623 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:54.069959 21624 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0da9e55527f4c899ab181e6f75b7f08. Latest consensus state: current_term: 1 leader_uuid: "f0da9e55527f4c899ab181e6f75b7f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0da9e55527f4c899ab181e6f75b7f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37679 } } }
I20260620 14:29:54.070041 21624 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08 [sys.catalog]: This master's current role is: LEADER
W20260620 14:29:54.073045 21637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:29:54.073101 21637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:29:54.073174 21631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:54.073796 21631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:54.078624 21631 catalog_manager.cc:1347] Generated new cluster ID: 7e152f9f73b54de4a62c510f4a22c54c
I20260620 14:29:54.078680 21631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:54.094331 21631 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:54.094646 21631 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:54.103531 21631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0da9e55527f4c899ab181e6f75b7f08: Generated new TSK 0
I20260620 14:29:54.104111 21631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:54.116210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 21544
2026-06-20T14:29:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:54 dist-test-slave-pk3j krb5kdc[21643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:54 dist-test-slave-pk3j krb5kdc[21643](info): set up 2 sockets
Jun 20 14:29:54 dist-test-slave-pk3j krb5kdc[21643](info): commencing operation
krb5kdc: starting...
W20260620 14:29:55.121373 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:55 dist-test-slave-pk3j krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:55.245798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:55.346257 21656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:55.346524 21656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:55.346601 21656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:55.350310 21656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:55.350390 21656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:55.350431 21656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:55.350487 21656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:55.350526 21656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:55.355242 21656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46279
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:55.356132 21656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:55.357007 21656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:55.364347 21664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:55.364387 21662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:55.364507 21661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:55.364566 21656 server_base.cc:1034] running on GCE node
I20260620 14:29:55.365120 21656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:55.366101 21656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:55.367255 21656 hybrid_clock.cc:648] HybridClock initialized: now 1781965795367239 us; error 29 us; skew 500 ppm
Jun 20 14:29:55 dist-test-slave-pk3j krb5kdc[21643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:55.369803 21656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:55.370571 21656 webserver.cc:460] Webserver started at http://127.0.0.1:44985/ using document root <none> and password file <none>
I20260620 14:29:55.371058 21656 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:55.371114 21656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:55.371258 21656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:55.372952 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d"
format_stamp: "Formatted at 2026-06-20 14:29:55 on dist-test-slave-pk3j"
server_key: "6c4aec6851ea121f79c5e2f82eae1e16"
server_key_iv: "6423f52615cf48a88938566170727f28"
server_key_version: "encryptionkey@0"
I20260620 14:29:55.373442 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d"
format_stamp: "Formatted at 2026-06-20 14:29:55 on dist-test-slave-pk3j"
server_key: "6c4aec6851ea121f79c5e2f82eae1e16"
server_key_iv: "6423f52615cf48a88938566170727f28"
server_key_version: "encryptionkey@0"
I20260620 14:29:55.376609 21656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:29:55.378710 21671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:55.379771 21656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:29:55.379856 21656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d"
format_stamp: "Formatted at 2026-06-20 14:29:55 on dist-test-slave-pk3j"
server_key: "6c4aec6851ea121f79c5e2f82eae1e16"
server_key_iv: "6423f52615cf48a88938566170727f28"
server_key_version: "encryptionkey@0"
I20260620 14:29:55.379930 21656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:55.392571 21656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:55.394938 21656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:55.395063 21656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:55.401096 21656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46279
I20260620 14:29:55.401172 21723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46279 every 8 connection(s)
I20260620 14:29:55.401944 21656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:55.404266 21724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:55.408871 21724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d: Bootstrap starting.
I20260620 14:29:55.410930 21724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:55.410962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 21656
I20260620 14:29:55.411101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:55.411376 392 external_mini_cluster.cc:1428] Setting key 4660c6427bc0383553efc8d20484343c
I20260620 14:29:55.411800 21724 log.cc:824] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:55.413901 21724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d: No bootstrap required, opened a new log
I20260620 14:29:55.418032 21724 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } }
I20260620 14:29:55.418270 21724 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:55.418329 21724 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d95c68dbc9422bac5ca14a2f57bb2d, State: Initialized, Role: FOLLOWER
I20260620 14:29:55.418761 21724 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } }
I20260620 14:29:55.418879 21724 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:55.418968 21724 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:55.419092 21724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:55.419860 21724 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } }
Jun 20 14:29:55 dist-test-slave-pk3j krb5kdc[21643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:29:55.420209 21724 leader_election.cc:302] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2d95c68dbc9422bac5ca14a2f57bb2d; no voters:
I20260620 14:29:55.420506 21724 leader_election.cc:288] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:55.420604 21729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:55.420828 21729 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [term 1 LEADER]: Becoming Leader. State: Replica: d2d95c68dbc9422bac5ca14a2f57bb2d, State: Running, Role: LEADER
I20260620 14:29:55.421145 21729 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } }
I20260620 14:29:55.421730 21724 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:55.423448 21730 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } } }
I20260620 14:29:55.423588 21730 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:55.423846 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2d95c68dbc9422bac5ca14a2f57bb2d. Latest consensus state: current_term: 1 leader_uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d95c68dbc9422bac5ca14a2f57bb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46279 } } }
I20260620 14:29:55.424022 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:55.423928 21744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:55.426333 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:55.412398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:44301 (local address 127.0.0.1:46279)
0620 14:29:55.412811 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:55.412828 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:55.412864 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:29:55.413628 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:55.413650 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:55.414027 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:55.414352 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:55.415019 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:55.416487 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:55.417327 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:55.417982 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:55.418137 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:55.420756 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:55.420765 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:55.420784 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:55.420818 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:55.422869 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:55.423454 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:55.423460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:55.423465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:55.423597 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:55.424275 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:55.424278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:55.424280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:55.424722 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:55.424973 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:55.425281 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 14:29:55.425499 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":115,"threads_started":1}
I20260620 14:29:55.427120 21744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:55.429289 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:55.432641 21744 catalog_manager.cc:1347] Generated new cluster ID: d3418283fdbf4ad698ebe877a5c06021
I20260620 14:29:55.432700 21744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:55.438755 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:55.431145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:41505 (local address 127.0.0.1:46279)
0620 14:29:55.431230 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:55.431234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:55.431498 (+ 264us) server_negotiation.cc:407] Connection header received
0620 14:29:55.431617 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:55.431620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:55.431657 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:55.431742 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:55.432374 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:55.432933 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:55.433769 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:55.434010 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:55.434111 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:55.435421 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:55.435444 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:55.435446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:55.435471 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:55.437044 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:55.437551 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:55.437554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:55.437556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:55.437634 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:55.437947 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:55.437950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:55.437951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:55.438106 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:55.438251 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:55.438520 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 14:29:55.438603 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:29:55.469875 21744 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:55.471009 21744 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:55.487809 21744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2d95c68dbc9422bac5ca14a2f57bb2d: Generated new TSK 0
I20260620 14:29:55.488478 21744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:55.495199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 21656
W20260620 14:29:55.499801 21641 connection.cc:535] client connection to 127.0.0.1:46279 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46279): Connection reset by peer (error 104)
2026-06-20T14:29:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:55 dist-test-slave-pk3j krb5kdc[21755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:55 dist-test-slave-pk3j krb5kdc[21755](info): set up 2 sockets
Jun 20 14:29:55 dist-test-slave-pk3j krb5kdc[21755](info): commencing operation
krb5kdc: starting...
W20260620 14:29:56.513423 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:56 dist-test-slave-pk3j krb5kdc[21755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:56.623725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:56.723985 21768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:56.724195 21768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:56.724236 21768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:56.728096 21768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:56.728140 21768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:56.728158 21768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:56.728204 21768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:56.728236 21768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:56.732933 21768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42337
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:56.733827 21768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:56.734711 21768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:56.742151 21774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:56.742151 21776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:56.742159 21773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:56.742506 21768 server_base.cc:1034] running on GCE node
I20260620 14:29:56.742941 21768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:56.743877 21768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:56.745038 21768 hybrid_clock.cc:648] HybridClock initialized: now 1781965796745021 us; error 30 us; skew 500 ppm
Jun 20 14:29:56 dist-test-slave-pk3j krb5kdc[21755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:56.747689 21768 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:56.748351 21768 webserver.cc:460] Webserver started at http://127.0.0.1:39777/ using document root <none> and password file <none>
I20260620 14:29:56.748803 21768 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:56.748855 21768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:56.748994 21768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:56.751183 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "593c64dda22941caaab3afb420523430"
format_stamp: "Formatted at 2026-06-20 14:29:56 on dist-test-slave-pk3j"
server_key: "52714863296b340556af525087e043dd"
server_key_iv: "35eec064fa301049ebd6a0376d0225f9"
server_key_version: "encryptionkey@0"
I20260620 14:29:56.751674 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "593c64dda22941caaab3afb420523430"
format_stamp: "Formatted at 2026-06-20 14:29:56 on dist-test-slave-pk3j"
server_key: "52714863296b340556af525087e043dd"
server_key_iv: "35eec064fa301049ebd6a0376d0225f9"
server_key_version: "encryptionkey@0"
I20260620 14:29:56.754900 21768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:29:56.756973 21783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:56.758131 21768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:56.758258 21768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "593c64dda22941caaab3afb420523430"
format_stamp: "Formatted at 2026-06-20 14:29:56 on dist-test-slave-pk3j"
server_key: "52714863296b340556af525087e043dd"
server_key_iv: "35eec064fa301049ebd6a0376d0225f9"
server_key_version: "encryptionkey@0"
I20260620 14:29:56.758359 21768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:56.770780 21768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:56.773119 21768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:56.773329 21768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:56.779531 21768 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42337
I20260620 14:29:56.779597 21835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42337 every 8 connection(s)
I20260620 14:29:56.780354 21768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:56.782701 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:56.787475 21836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430: Bootstrap starting.
I20260620 14:29:56.788313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 21768
I20260620 14:29:56.788473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:56.788718 392 external_mini_cluster.cc:1428] Setting key 785b624903411e2f7c85787aadca69f7
I20260620 14:29:56.789958 21836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:56.790580 21836 log.cc:824] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:56.792200 21836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430: No bootstrap required, opened a new log
I20260620 14:29:56.795531 21836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593c64dda22941caaab3afb420523430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42337 } }
I20260620 14:29:56.795758 21836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:56.795825 21836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c64dda22941caaab3afb420523430, State: Initialized, Role: FOLLOWER
I20260620 14:29:56.796249 21836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593c64dda22941caaab3afb420523430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42337 } }
I20260620 14:29:56.796396 21836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:56.796481 21836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:56.796594 21836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:56.796838 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:56.789669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:37227 (local address 127.0.0.1:42337)
0620 14:29:56.790016 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:56.790035 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:56.790090 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:29:56.790617 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:56.790631 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:56.791009 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:56.791332 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:56.791698 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:56.792579 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:56.793557 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:56.794009 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:56.794178 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:56.795002 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:56.795007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:56.795017 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:56.795344 (+ 327us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:56.795390 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:56.795435 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:56.795574 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:56.795927 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 14:29:56.796040 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":112,"threads_started":1}
I20260620 14:29:56.797492 21836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593c64dda22941caaab3afb420523430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42337 } }
I20260620 14:29:56.797837 21836 leader_election.cc:302] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 593c64dda22941caaab3afb420523430; no voters:
I20260620 14:29:56.798136 21836 leader_election.cc:288] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:56.798228 21841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:56.798439 21841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [term 1 LEADER]: Becoming Leader. State: Replica: 593c64dda22941caaab3afb420523430, State: Running, Role: LEADER
I20260620 14:29:56.798758 21841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593c64dda22941caaab3afb420523430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42337 } }
I20260620 14:29:56.799212 21836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:56.799765 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:56.800406 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "593c64dda22941caaab3afb420523430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593c64dda22941caaab3afb420523430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42337 } } }
I20260620 14:29:56.800380 21843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 593c64dda22941caaab3afb420523430. Latest consensus state: current_term: 1 leader_uuid: "593c64dda22941caaab3afb420523430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "593c64dda22941caaab3afb420523430" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42337 } } }
I20260620 14:29:56.800508 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:56.800508 21843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430 [sys.catalog]: This master's current role is: LEADER
W20260620 14:29:56.803812 21860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:29:56.803894 21860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:29:56.803969 21850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:56.804642 21850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:56.806272 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:56.802181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:59399 (local address 127.0.0.1:42337)
0620 14:29:56.802279 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:56.802283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:56.802295 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:29:56.802327 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:56.802329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:56.802370 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:56.802449 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:56.802793 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:56.803491 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:56.804517 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:56.804829 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:56.804932 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:56.805798 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:56.805801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:56.805804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:56.805816 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:56.805840 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:56.805852 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:56.805946 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:56.806116 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:29:56.806145 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:29:56.809456 21850 catalog_manager.cc:1347] Generated new cluster ID: 8b3ad24ef09a463db952f5aee409d6a9
I20260620 14:29:56.809511 21850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:56.827888 21850 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:56.829047 21850 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:56.834861 21850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 593c64dda22941caaab3afb420523430: Generated new TSK 0
I20260620 14:29:56.835513 21850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:56.841874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 21768
W20260620 14:29:56.847149 21753 connection.cc:535] client connection to 127.0.0.1:42337 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42337): Connection reset by peer (error 104)
2026-06-20T14:29:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:56 dist-test-slave-pk3j krb5kdc[21867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:56 dist-test-slave-pk3j krb5kdc[21867](info): set up 2 sockets
Jun 20 14:29:56 dist-test-slave-pk3j krb5kdc[21867](info): commencing operation
krb5kdc: starting...
W20260620 14:29:57.837392 392 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:57 dist-test-slave-pk3j krb5kdc[21867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:29:57.945540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:58.048560 21880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:58.048754 21880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:58.048789 21880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:58.052768 21880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:58.052816 21880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:58.052835 21880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:58.052850 21880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:58.052901 21880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:58.057755 21880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:58.058612 21880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:58.059459 21880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:29:58.067217 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:58.067221 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:58.067417 21880 server_base.cc:1034] running on GCE node
W20260620 14:29:58.067461 21885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:58.067924 21880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:58.068877 21880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:58.070061 21880 hybrid_clock.cc:648] HybridClock initialized: now 1781965798070038 us; error 36 us; skew 500 ppm
Jun 20 14:29:58 dist-test-slave-pk3j krb5kdc[21867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:58.081277 21880 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:29:58.082090 21880 webserver.cc:460] Webserver started at http://127.0.0.1:41105/ using document root <none> and password file <none>
I20260620 14:29:58.082574 21880 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:58.082655 21880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:58.082854 21880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:58.084625 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "dc9afa37c7ad447a892c231a3b67a483"
format_stamp: "Formatted at 2026-06-20 14:29:58 on dist-test-slave-pk3j"
server_key: "5e10aa2a392a8ce8f6fc0c1fdd9d7925"
server_key_iv: "4bef0b4cde6c2a78b6509d457cdeca17"
server_key_version: "encryptionkey@0"
I20260620 14:29:58.085121 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc9afa37c7ad447a892c231a3b67a483"
format_stamp: "Formatted at 2026-06-20 14:29:58 on dist-test-slave-pk3j"
server_key: "5e10aa2a392a8ce8f6fc0c1fdd9d7925"
server_key_iv: "4bef0b4cde6c2a78b6509d457cdeca17"
server_key_version: "encryptionkey@0"
I20260620 14:29:58.088449 21880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:29:58.090706 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:58.091800 21880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:29:58.091928 21880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "dc9afa37c7ad447a892c231a3b67a483"
format_stamp: "Formatted at 2026-06-20 14:29:58 on dist-test-slave-pk3j"
server_key: "5e10aa2a392a8ce8f6fc0c1fdd9d7925"
server_key_iv: "4bef0b4cde6c2a78b6509d457cdeca17"
server_key_version: "encryptionkey@0"
I20260620 14:29:58.092029 21880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:58.096119 21880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:58.098706 21880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:58.098888 21880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:58.105113 21880 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37423
I20260620 14:29:58.105124 21947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37423 every 8 connection(s)
I20260620 14:29:58.105931 21880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:58.108374 21948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:58.110711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 21880
I20260620 14:29:58.110858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:58.111127 392 external_mini_cluster.cc:1428] Setting key 743a80001300a6c2dcd62635f7b7530f
I20260620 14:29:58.114063 21948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483: Bootstrap starting.
I20260620 14:29:58.116178 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:58.112358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:53027 (local address 127.0.0.1:37423)
0620 14:29:58.112746 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:58.112764 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:58.112803 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:29:58.113571 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:58.113591 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:58.113902 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:58.114245 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:58.114867 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:58.114872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:58.114885 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:58.114896 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:58.114952 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:58.114992 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:58.115138 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:58.115237 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 14:29:58.115378 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
I20260620 14:29:58.116591 21948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:58.117411 21948 log.cc:824] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483: Log is configured to *not* fsync() on all Append() calls
I20260620 14:29:58.118885 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:58.119422 21948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483: No bootstrap required, opened a new log
I20260620 14:29:58.122360 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:58.120936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.142:60477 (local address 127.0.0.1:37423)
0620 14:29:58.121048 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:58.121052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:58.121191 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:29:58.121328 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:58.121330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:58.121355 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:58.121415 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:58.122005 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:58.122008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:58.122010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:29:58.122019 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:58.122039 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:29:58.122048 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:58.122112 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:58.122222 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 14:29:58.122248 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:29:58.123862 21948 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9afa37c7ad447a892c231a3b67a483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } }
I20260620 14:29:58.124070 21948 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:58.124123 21948 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9afa37c7ad447a892c231a3b67a483, State: Initialized, Role: FOLLOWER
I20260620 14:29:58.124483 21948 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9afa37c7ad447a892c231a3b67a483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } }
I20260620 14:29:58.124624 21948 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:58.124684 21948 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:58.124770 21948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:58.125716 21948 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9afa37c7ad447a892c231a3b67a483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } }
I20260620 14:29:58.126029 21948 leader_election.cc:302] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc9afa37c7ad447a892c231a3b67a483; no voters:
I20260620 14:29:58.126309 21948 leader_election.cc:288] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:58.126415 21958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:58.126649 21958 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [term 1 LEADER]: Becoming Leader. State: Replica: dc9afa37c7ad447a892c231a3b67a483, State: Running, Role: LEADER
I20260620 14:29:58.126935 21958 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9afa37c7ad447a892c231a3b67a483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } }
I20260620 14:29:58.127450 21948 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:58.128000 21958 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9afa37c7ad447a892c231a3b67a483. Latest consensus state: current_term: 1 leader_uuid: "dc9afa37c7ad447a892c231a3b67a483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9afa37c7ad447a892c231a3b67a483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } } }
I20260620 14:29:58.128109 21958 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:58.128456 21959 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc9afa37c7ad447a892c231a3b67a483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9afa37c7ad447a892c231a3b67a483" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37423 } } }
I20260620 14:29:58.128584 21959 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:58.128472 21965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:58.131650 21965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:58.136616 21965 catalog_manager.cc:1347] Generated new cluster ID: 90e9e0fa78654f2faade45f258e52bf7
I20260620 14:29:58.136721 21965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:58.149263 21965 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:58.149519 21965 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:58.154937 21965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc9afa37c7ad447a892c231a3b67a483: Generated new TSK 0
I20260620 14:29:58.155494 21965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:58.163280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 21880
2026-06-20T14:29:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:58 dist-test-slave-pk3j krb5kdc[21979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:58 dist-test-slave-pk3j krb5kdc[21979](info): set up 2 sockets
Jun 20 14:29:58 dist-test-slave-pk3j krb5kdc[21979](info): commencing operation
krb5kdc: starting...
W20260620 14:29:59.173264 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:29:59 dist-test-slave-pk3j krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:29:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:29:59.281395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:29:59.383515 21992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:29:59.383761 21992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:29:59.383857 21992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:29:59.387796 21992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:29:59.387873 21992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:29:59.387910 21992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:29:59.387966 21992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:29:59.388001 21992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:29:59.392856 21992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:29:59.393790 21992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:29:59.394680 21992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:29:59.402478 21992 server_base.cc:1034] running on GCE node
W20260620 14:29:59.402436 21998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:59.402412 22000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:29:59.402421 21997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:29:59.403119 21992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:29:59.404088 21992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:29:59.405268 21992 hybrid_clock.cc:648] HybridClock initialized: now 1781965799405240 us; error 39 us; skew 500 ppm
Jun 20 14:29:59 dist-test-slave-pk3j krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:29:59.407708 21992 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:29:59.408457 21992 webserver.cc:460] Webserver started at http://127.0.98.62:42585/ using document root <none> and password file <none>
I20260620 14:29:59.408953 21992 fs_manager.cc:359] Metadata directory not provided
I20260620 14:29:59.409039 21992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:29:59.409302 21992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:29:59.411099 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "52e80dcaab5a4541a78df59c00b2dc60"
format_stamp: "Formatted at 2026-06-20 14:29:59 on dist-test-slave-pk3j"
server_key: "94168dcfdf44e90cb4af73e054595ef4"
server_key_iv: "7ca13d4d1808e27ae2d95753367f6dcd"
server_key_version: "encryptionkey@0"
I20260620 14:29:59.411626 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "52e80dcaab5a4541a78df59c00b2dc60"
format_stamp: "Formatted at 2026-06-20 14:29:59 on dist-test-slave-pk3j"
server_key: "94168dcfdf44e90cb4af73e054595ef4"
server_key_iv: "7ca13d4d1808e27ae2d95753367f6dcd"
server_key_version: "encryptionkey@0"
I20260620 14:29:59.414989 21992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:29:59.417126 22007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:29:59.418270 21992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:29:59.418399 21992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "52e80dcaab5a4541a78df59c00b2dc60"
format_stamp: "Formatted at 2026-06-20 14:29:59 on dist-test-slave-pk3j"
server_key: "94168dcfdf44e90cb4af73e054595ef4"
server_key_iv: "7ca13d4d1808e27ae2d95753367f6dcd"
server_key_version: "encryptionkey@0"
I20260620 14:29:59.418501 21992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:29:59.432250 21992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:29:59.434713 21992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:29:59.434896 21992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:29:59.441107 21992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34245
I20260620 14:29:59.441123 22059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34245 every 8 connection(s)
I20260620 14:29:59.441962 21992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:29:59.444404 22060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:29:59.446050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 21992
I20260620 14:29:59.446189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:29:59.446471 392 external_mini_cluster.cc:1428] Setting key be3ca7e5f56ec3269e8559ca7e7374de
I20260620 14:29:59.450412 22060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60: Bootstrap starting.
I20260620 14:29:59.452834 22060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60: Neither blocks nor log segments found. Creating new log.
I20260620 14:29:59.453733 22060 log.cc:824] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:29:59 dist-test-slave-pk3j krb5kdc[21979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:29:59.455586 22060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60: No bootstrap required, opened a new log
I20260620 14:29:59.460175 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:59.447628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46467 (local address 127.0.98.62:34245)
0620 14:29:59.447982 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:59.447996 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:59.448028 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:29:59.448592 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:59.448607 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:59.448897 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:59.449143 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:59.450005 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:59.450979 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:59.452147 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:59.452674 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:59.452783 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:59.455365 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:59.455373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:59.455397 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:59.455425 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:59.457323 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:59.457795 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:59.457803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:59.457810 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:59.457964 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:59.458268 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:59.458271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:59.458273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:59.458710 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:59.458923 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:59.459129 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:29:59.459333 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":118,"threads_started":1}
I20260620 14:29:59.460032 22060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e80dcaab5a4541a78df59c00b2dc60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260620 14:29:59.460268 22060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:29:59.460310 22060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e80dcaab5a4541a78df59c00b2dc60, State: Initialized, Role: FOLLOWER
I20260620 14:29:59.460719 22060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e80dcaab5a4541a78df59c00b2dc60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260620 14:29:59.460862 22060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:29:59.460929 22060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:29:59.461021 22060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:29:59.461856 22060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e80dcaab5a4541a78df59c00b2dc60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260620 14:29:59.462160 22060 leader_election.cc:302] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e80dcaab5a4541a78df59c00b2dc60; no voters:
I20260620 14:29:59.462467 22060 leader_election.cc:288] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:29:59.462548 22065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:29:59.462739 22065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [term 1 LEADER]: Becoming Leader. State: Replica: 52e80dcaab5a4541a78df59c00b2dc60, State: Running, Role: LEADER
I20260620 14:29:59.462915 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:29:59.463004 22065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e80dcaab5a4541a78df59c00b2dc60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } }
I20260620 14:29:59.463644 22060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:29:59.464478 22067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e80dcaab5a4541a78df59c00b2dc60. Latest consensus state: current_term: 1 leader_uuid: "52e80dcaab5a4541a78df59c00b2dc60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e80dcaab5a4541a78df59c00b2dc60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } } }
I20260620 14:29:59.464447 22066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e80dcaab5a4541a78df59c00b2dc60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e80dcaab5a4541a78df59c00b2dc60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34245 } } }
I20260620 14:29:59.464579 22067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:59.464579 22066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60 [sys.catalog]: This master's current role is: LEADER
I20260620 14:29:59.465188 22078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:29:59.467711 22078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:29:59.473081 22078 catalog_manager.cc:1347] Generated new cluster ID: d9ae0aeb00ee41f2bc8009af6c579a71
I20260620 14:29:59.473142 22078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:29:59.476761 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:29:59.465189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47179 (local address 127.0.98.62:34245)
0620 14:29:59.465320 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:29:59.465324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:29:59.469385 (+ 4061us) server_negotiation.cc:407] Connection header received
0620 14:29:59.469496 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:29:59.469499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:29:59.469535 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:29:59.469621 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:29:59.470205 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:59.470713 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:59.471585 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:29:59.471879 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:29:59.471936 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:29:59.473498 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:29:59.473502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:29:59.473504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:29:59.473527 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:29:59.475022 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:59.475523 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:59.475526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:59.475528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:59.475604 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:29:59.475934 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:29:59.475937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:29:59.475939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:29:59.476099 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:29:59.476202 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:29:59.476548 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 14:29:59.476628 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:29:59.482911 22078 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:29:59.483915 22078 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:29:59.490518 22078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e80dcaab5a4541a78df59c00b2dc60: Generated new TSK 0
I20260620 14:29:59.491089 22078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:29:59.497812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 21992
W20260620 14:29:59.502912 21977 connection.cc:535] client connection to 127.0.98.62:34245 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34245): Connection reset by peer (error 104)
2026-06-20T14:29:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:29:59 dist-test-slave-pk3j krb5kdc[22091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:29:59 dist-test-slave-pk3j krb5kdc[22091](info): set up 2 sockets
Jun 20 14:29:59 dist-test-slave-pk3j krb5kdc[22091](info): commencing operation
krb5kdc: starting...
W20260620 14:30:00.505070 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:00 dist-test-slave-pk3j krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:00.609817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:00.708751 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:00.708983 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:00.709048 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:00.712648 22104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:00.712745 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:00.712800 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:00.712865 22104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:00.712899 22104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:00.717633 22104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:00.718463 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:00.719310 22104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:00.726787 22112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:00.726825 22109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:00.726821 22110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:00.726945 22104 server_base.cc:1034] running on GCE node
I20260620 14:30:00.727530 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:00.728451 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:00.729633 22104 hybrid_clock.cc:648] HybridClock initialized: now 1781965800729616 us; error 33 us; skew 500 ppm
Jun 20 14:30:00 dist-test-slave-pk3j krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:00.732123 22104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:00.732854 22104 webserver.cc:460] Webserver started at http://127.0.98.62:35279/ using document root <none> and password file <none>
I20260620 14:30:00.733390 22104 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:00.733450 22104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:00.733600 22104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:00.735438 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "ba1a01b1ca1945a386682d93c462c37a"
format_stamp: "Formatted at 2026-06-20 14:30:00 on dist-test-slave-pk3j"
server_key: "37a1eba657ed133476338bcc983d49de"
server_key_iv: "bdff8653e4a3803f4e38462d96f5918b"
server_key_version: "encryptionkey@0"
I20260620 14:30:00.735920 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba1a01b1ca1945a386682d93c462c37a"
format_stamp: "Formatted at 2026-06-20 14:30:00 on dist-test-slave-pk3j"
server_key: "37a1eba657ed133476338bcc983d49de"
server_key_iv: "bdff8653e4a3803f4e38462d96f5918b"
server_key_version: "encryptionkey@0"
I20260620 14:30:00.739221 22104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:30:00.741256 22119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:00.742288 22104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:30:00.742413 22104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "ba1a01b1ca1945a386682d93c462c37a"
format_stamp: "Formatted at 2026-06-20 14:30:00 on dist-test-slave-pk3j"
server_key: "37a1eba657ed133476338bcc983d49de"
server_key_iv: "bdff8653e4a3803f4e38462d96f5918b"
server_key_version: "encryptionkey@0"
I20260620 14:30:00.742502 22104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:00.754438 22104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:00.756995 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:00.757117 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:00.763023 22104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36199
I20260620 14:30:00.763089 22171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36199 every 8 connection(s)
I20260620 14:30:00.763794 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:00.764822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 22104
I20260620 14:30:00.764961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:00.765252 392 external_mini_cluster.cc:1428] Setting key 1d8bc18c7dc7391e5c19a1e6b21763f4
I20260620 14:30:00.766558 22172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:00.772737 22172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a: Bootstrap starting.
I20260620 14:30:00.774325 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:00.766516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53125 (local address 127.0.98.62:36199)
0620 14:30:00.766908 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:00.766925 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:00.766963 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:30:00.767714 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:00.767734 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:00.768088 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:00.768405 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:00.768744 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:00.770118 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:00.770957 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:00.771620 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:00.771773 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:00.772514 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:00.772520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:00.772533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:00.772971 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:00.773031 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:00.773058 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:00.773188 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:00.773367 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:30:00.773477 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":133,"threads_started":1}
I20260620 14:30:00.775172 22172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:00.775882 22172 log.cc:824] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:00.776933 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:00.777761 22172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a: No bootstrap required, opened a new log
I20260620 14:30:00.781697 22172 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1a01b1ca1945a386682d93c462c37a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } }
I20260620 14:30:00.781903 22172 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:00.781956 22172 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1a01b1ca1945a386682d93c462c37a, State: Initialized, Role: FOLLOWER
I20260620 14:30:00.782384 22172 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1a01b1ca1945a386682d93c462c37a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } }
I20260620 14:30:00.782532 22172 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:00.782598 22172 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:00.782704 22172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:00.782981 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:00.778822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39443 (local address 127.0.98.62:36199)
0620 14:30:00.778919 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:00.778922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:00.779048 (+ 126us) server_negotiation.cc:407] Connection header received
0620 14:30:00.779132 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:00.779134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:00.779176 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:00.779245 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:00.779456 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:00.780251 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:00.780994 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:00.781359 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:00.781460 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:00.782268 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:00.782291 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:00.782296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:00.782313 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:00.782338 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:00.782351 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:00.782416 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:00.782698 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:30:00.782804 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:30:00.783677 22172 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1a01b1ca1945a386682d93c462c37a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } }
I20260620 14:30:00.783967 22172 leader_election.cc:302] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba1a01b1ca1945a386682d93c462c37a; no voters:
I20260620 14:30:00.784209 22172 leader_election.cc:288] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:00.784298 22182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:00.784505 22182 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [term 1 LEADER]: Becoming Leader. State: Replica: ba1a01b1ca1945a386682d93c462c37a, State: Running, Role: LEADER
I20260620 14:30:00.784771 22182 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1a01b1ca1945a386682d93c462c37a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } }
I20260620 14:30:00.785274 22172 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:00.786469 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba1a01b1ca1945a386682d93c462c37a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1a01b1ca1945a386682d93c462c37a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } }
I20260620 14:30:00.786487 22184 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba1a01b1ca1945a386682d93c462c37a. Latest consensus state: current_term: 1 leader_uuid: "ba1a01b1ca1945a386682d93c462c37a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba1a01b1ca1945a386682d93c462c37a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } }
I20260620 14:30:00.786650 22184 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:00.786650 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:00.787156 22192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:00.789539 22192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:00.794396 22192 catalog_manager.cc:1347] Generated new cluster ID: 5cb7226a12354288b179a38851abe283
I20260620 14:30:00.794458 22192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:00.809942 22192 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:00.811020 22192 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:00.820353 22192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba1a01b1ca1945a386682d93c462c37a: Generated new TSK 0
I20260620 14:30:00.820922 22192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:00.826188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 22104
W20260620 14:30:00.831256 22089 connection.cc:535] client connection to 127.0.98.62:36199 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36199): Connection reset by peer (error 104)
2026-06-20T14:30:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:00 dist-test-slave-pk3j krb5kdc[22203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:00 dist-test-slave-pk3j krb5kdc[22203](info): set up 2 sockets
Jun 20 14:30:00 dist-test-slave-pk3j krb5kdc[22203](info): commencing operation
krb5kdc: starting...
W20260620 14:30:01.817342 392 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:01 dist-test-slave-pk3j krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:01.923697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:02.026669 22216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:02.026927 22216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:02.027001 22216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:02.030772 22216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:02.030849 22216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:02.030887 22216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:02.030922 22216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:02.030959 22216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:02.035770 22216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:02.036666 22216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:02.037572 22216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:02.045056 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.045136 22222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:02.045058 22224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:02.045287 22216 server_base.cc:1034] running on GCE node
I20260620 14:30:02.045795 22216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:02.046978 22216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:02.048146 22216 hybrid_clock.cc:648] HybridClock initialized: now 1781965802048082 us; error 76 us; skew 500 ppm
Jun 20 14:30:02 dist-test-slave-pk3j krb5kdc[22203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:02.051064 22216 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:02.051836 22216 webserver.cc:460] Webserver started at http://127.0.98.62:45807/ using document root <none> and password file <none>
I20260620 14:30:02.052296 22216 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:02.052350 22216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:02.052497 22216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:02.054347 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "2ff3077418024536b5c401d9d73fb8ed"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-pk3j"
server_key: "050765950753e9e5113eb416e0695109"
server_key_iv: "d40755b217999a105f5edcea83a09019"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.054800 22216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ff3077418024536b5c401d9d73fb8ed"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-pk3j"
server_key: "050765950753e9e5113eb416e0695109"
server_key_iv: "d40755b217999a105f5edcea83a09019"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.058094 22216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:30:02.060170 22231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:02.061276 22216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:02.061398 22216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "2ff3077418024536b5c401d9d73fb8ed"
format_stamp: "Formatted at 2026-06-20 14:30:02 on dist-test-slave-pk3j"
server_key: "050765950753e9e5113eb416e0695109"
server_key_iv: "d40755b217999a105f5edcea83a09019"
server_key_version: "encryptionkey@0"
I20260620 14:30:02.061493 22216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:02.065411 22216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:02.067896 22216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:02.068065 22216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:02.074271 22216 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46259
I20260620 14:30:02.074280 22283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46259 every 8 connection(s)
I20260620 14:30:02.075048 22216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:02.077526 22284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:02.078428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 22216
I20260620 14:30:02.078575 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:02.078814 392 external_mini_cluster.cc:1428] Setting key 2f2d4fbf2d79c3cf3b149e3cca437b23
I20260620 14:30:02.083313 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed: Bootstrap starting.
I20260620 14:30:02.083453 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.079930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58459 (local address 127.0.98.62:46259)
0620 14:30:02.080302 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.080320 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.080360 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:30:02.081078 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.081097 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.081429 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.081739 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:02.082277 (+ 538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:02.082282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:02.082295 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:02.082306 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:02.082357 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:02.082379 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:02.082504 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.082564 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.082687 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":101,"threads_started":1}
I20260620 14:30:02.085661 22284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:02.085999 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:02.086416 22284 log.cc:824] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:02.088349 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed: No bootstrap required, opened a new log
I20260620 14:30:02.089285 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:02.087819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40329 (local address 127.0.98.62:46259)
0620 14:30:02.087934 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:02.087938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:02.088156 (+ 218us) server_negotiation.cc:407] Connection header received
0620 14:30:02.088261 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:02.088263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:02.088287 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:02.088348 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:02.088931 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:02.088934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:02.088936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:02.088945 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:02.088965 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:02.088974 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:02.089038 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:02.089129 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:30:02.089154 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:30:02.091989 22284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3077418024536b5c401d9d73fb8ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } }
I20260620 14:30:02.092188 22284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:02.092249 22284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff3077418024536b5c401d9d73fb8ed, State: Initialized, Role: FOLLOWER
I20260620 14:30:02.092582 22284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3077418024536b5c401d9d73fb8ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } }
I20260620 14:30:02.092684 22284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:02.092725 22284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:02.092787 22284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:02.093642 22284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3077418024536b5c401d9d73fb8ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } }
I20260620 14:30:02.093940 22284 leader_election.cc:302] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ff3077418024536b5c401d9d73fb8ed; no voters:
I20260620 14:30:02.094192 22284 leader_election.cc:288] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:02.094295 22294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:02.094507 22294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [term 1 LEADER]: Becoming Leader. State: Replica: 2ff3077418024536b5c401d9d73fb8ed, State: Running, Role: LEADER
I20260620 14:30:02.094769 22294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3077418024536b5c401d9d73fb8ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } }
I20260620 14:30:02.095196 22284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:02.096220 22295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ff3077418024536b5c401d9d73fb8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3077418024536b5c401d9d73fb8ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } } }
I20260620 14:30:02.096340 22295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:02.096638 22296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ff3077418024536b5c401d9d73fb8ed. Latest consensus state: current_term: 1 leader_uuid: "2ff3077418024536b5c401d9d73fb8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3077418024536b5c401d9d73fb8ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46259 } } }
I20260620 14:30:02.096724 22296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:02.096812 22303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:02.099377 22303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:02.104190 22303 catalog_manager.cc:1347] Generated new cluster ID: 79ce76c2c0064ce28c353beace35a262
I20260620 14:30:02.104247 22303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:02.116219 22303 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:02.116484 22303 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:02.123337 22303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ff3077418024536b5c401d9d73fb8ed: Generated new TSK 0
I20260620 14:30:02.123926 22303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:02.136391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 22216
2026-06-20T14:30:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:02 dist-test-slave-pk3j krb5kdc[22315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:02 dist-test-slave-pk3j krb5kdc[22315](info): set up 2 sockets
Jun 20 14:30:02 dist-test-slave-pk3j krb5kdc[22315](info): commencing operation
krb5kdc: starting...
W20260620 14:30:03.125414 392 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:03 dist-test-slave-pk3j krb5kdc[22315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:03.231709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36879
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:03.335817 22328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:03.336045 22328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:03.336086 22328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:03.339896 22328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:03.339941 22328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:03.339974 22328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:03.339989 22328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:03.340003 22328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:03.344727 22328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:03.345620 22328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:03.346482 22328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:03.354036 22333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:03.354020 22334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:03.354012 22336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:03.354194 22328 server_base.cc:1034] running on GCE node
I20260620 14:30:03.354799 22328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:03.355729 22328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:03.356913 22328 hybrid_clock.cc:648] HybridClock initialized: now 1781965803356905 us; error 28 us; skew 500 ppm
Jun 20 14:30:03 dist-test-slave-pk3j krb5kdc[22315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:03.363013 22328 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:03.363760 22328 webserver.cc:460] Webserver started at http://127.0.98.62:42911/ using document root <none> and password file <none>
I20260620 14:30:03.364260 22328 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:03.364341 22328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:03.364542 22328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:03.366408 22328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "4078e264108a4703bb2df59e756bfc74"
format_stamp: "Formatted at 2026-06-20 14:30:03 on dist-test-slave-pk3j"
server_key: "35903d852c0389acf1fd082fa4debb91"
server_key_iv: "23402cb26bf82f05a6e25aa3f146c6d5"
server_key_version: "encryptionkey@0"
I20260620 14:30:03.366930 22328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "4078e264108a4703bb2df59e756bfc74"
format_stamp: "Formatted at 2026-06-20 14:30:03 on dist-test-slave-pk3j"
server_key: "35903d852c0389acf1fd082fa4debb91"
server_key_iv: "23402cb26bf82f05a6e25aa3f146c6d5"
server_key_version: "encryptionkey@0"
I20260620 14:30:03.370203 22328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:30:03.372330 22343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:03.373450 22328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:30:03.373567 22328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "4078e264108a4703bb2df59e756bfc74"
format_stamp: "Formatted at 2026-06-20 14:30:03 on dist-test-slave-pk3j"
server_key: "35903d852c0389acf1fd082fa4debb91"
server_key_iv: "23402cb26bf82f05a6e25aa3f146c6d5"
server_key_version: "encryptionkey@0"
I20260620 14:30:03.373663 22328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:03.383721 22328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:03.386153 22328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:03.386328 22328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:03.392464 22328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36879
I20260620 14:30:03.392511 22395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36879 every 8 connection(s)
I20260620 14:30:03.393313 22328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:03.395617 22396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:03.396270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 22328
I20260620 14:30:03.396415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:03.396677 392 external_mini_cluster.cc:1428] Setting key 1fba17af0629a386dbd722058ef491bb
I20260620 14:30:03.401248 22396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74: Bootstrap starting.
I20260620 14:30:03.403641 22396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:03.404498 22396 log.cc:824] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:30:03 dist-test-slave-pk3j krb5kdc[22315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:03.406157 22396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74: No bootstrap required, opened a new log
I20260620 14:30:03.409341 22396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4078e264108a4703bb2df59e756bfc74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } }
I20260620 14:30:03.409473 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.397859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57107 (local address 127.0.98.62:36879)
0620 14:30:03.398204 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:03.398224 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:03.398254 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:30:03.398846 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:03.398861 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:03.399140 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:03.399394 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:03.400227 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.401092 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.402253 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.402676 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.402789 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.405381 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:03.405386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:03.405396 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:03.405419 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:03.406982 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:03.407414 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:03.407418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:03.407422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:03.407519 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:03.407782 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:03.407785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:03.407787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:03.408135 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:03.408324 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:03.408555 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:30:03.408800 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":126,"threads_started":1}
I20260620 14:30:03.409556 22396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:03.409626 22396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4078e264108a4703bb2df59e756bfc74, State: Initialized, Role: FOLLOWER
I20260620 14:30:03.410069 22396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4078e264108a4703bb2df59e756bfc74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } }
I20260620 14:30:03.410214 22396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:03.410274 22396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:03.410362 22396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:03.411227 22396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4078e264108a4703bb2df59e756bfc74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } }
I20260620 14:30:03.411567 22396 leader_election.cc:302] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4078e264108a4703bb2df59e756bfc74; no voters:
I20260620 14:30:03.411859 22396 leader_election.cc:288] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:03.411921 22401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:03.412025 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:03.412082 22401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [term 1 LEADER]: Becoming Leader. State: Replica: 4078e264108a4703bb2df59e756bfc74, State: Running, Role: LEADER
I20260620 14:30:03.412407 22401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4078e264108a4703bb2df59e756bfc74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } }
I20260620 14:30:03.412943 22396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:03.413508 22401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4078e264108a4703bb2df59e756bfc74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4078e264108a4703bb2df59e756bfc74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } }
I20260620 14:30:03.413692 22401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:03.413905 22402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4078e264108a4703bb2df59e756bfc74. Latest consensus state: current_term: 1 leader_uuid: "4078e264108a4703bb2df59e756bfc74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4078e264108a4703bb2df59e756bfc74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36879 } } }
I20260620 14:30:03.414057 22402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:03.414165 22413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:03.417064 22413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:03.422859 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:03.414099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33701 (local address 127.0.98.62:36879)
0620 14:30:03.414193 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:03.414197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:03.414634 (+ 437us) server_negotiation.cc:407] Connection header received
0620 14:30:03.414836 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:03.414839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:03.414875 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:03.414956 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:03.415743 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.416217 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.417379 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:03.417703 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:03.417759 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:03.419605 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:03.419608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:03.419610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:03.419632 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:03.421126 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:03.421635 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:03.421638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:03.421639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:03.421724 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:03.422019 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:03.422025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:03.422028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:03.422220 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:03.422362 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:03.422607 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:30:03.422716 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:30:03.423962 22413 catalog_manager.cc:1347] Generated new cluster ID: f97684b3197f4217b47eda8442276cd1
I20260620 14:30:03.424022 22413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:03.443039 22413 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:03.444078 22413 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:03.452775 22413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4078e264108a4703bb2df59e756bfc74: Generated new TSK 0
I20260620 14:30:03.453457 22413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:03.463815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 22328
2026-06-20T14:30:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:03 dist-test-slave-pk3j krb5kdc[22427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:03 dist-test-slave-pk3j krb5kdc[22427](info): set up 2 sockets
Jun 20 14:30:03 dist-test-slave-pk3j krb5kdc[22427](info): commencing operation
krb5kdc: starting...
W20260620 14:30:04.477129 392 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:04 dist-test-slave-pk3j krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:04.583909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:04.682883 22440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:04.683104 22440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:04.683188 22440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:04.686987 22440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:04.687067 22440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:04.687105 22440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:04.687157 22440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:04.687197 22440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:04.691984 22440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:04.692813 22440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:04.693683 22440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:04.701368 22448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:04.701351 22445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:04.701378 22446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:04.701725 22440 server_base.cc:1034] running on GCE node
I20260620 14:30:04.702256 22440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:04.703202 22440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:04.704375 22440 hybrid_clock.cc:648] HybridClock initialized: now 1781965804704360 us; error 32 us; skew 500 ppm
Jun 20 14:30:04 dist-test-slave-pk3j krb5kdc[22427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:04.706925 22440 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:04.707732 22440 webserver.cc:460] Webserver started at http://127.0.98.62:45365/ using document root <none> and password file <none>
I20260620 14:30:04.708343 22440 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:04.708515 22440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:04.708726 22440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:04.710532 22440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "e583deafa2fb4cf2b3b6d909d963d298"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-pk3j"
server_key: "72b276cabc8902c07f60356ea5bd72b8"
server_key_iv: "fc1a7b3d70a9147240bad5e4c040149f"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.711069 22440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "e583deafa2fb4cf2b3b6d909d963d298"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-pk3j"
server_key: "72b276cabc8902c07f60356ea5bd72b8"
server_key_iv: "fc1a7b3d70a9147240bad5e4c040149f"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.714412 22440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:30:04.716672 22455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:04.717787 22440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:04.717912 22440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "e583deafa2fb4cf2b3b6d909d963d298"
format_stamp: "Formatted at 2026-06-20 14:30:04 on dist-test-slave-pk3j"
server_key: "72b276cabc8902c07f60356ea5bd72b8"
server_key_iv: "fc1a7b3d70a9147240bad5e4c040149f"
server_key_version: "encryptionkey@0"
I20260620 14:30:04.718010 22440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:04.735205 22440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:04.738113 22440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:04.738287 22440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:04.744560 22440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38889
I20260620 14:30:04.744583 22507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38889 every 8 connection(s)
I20260620 14:30:04.745383 22440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:04.747745 22508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:04.748626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 22440
I20260620 14:30:04.748768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:04.749030 392 external_mini_cluster.cc:1428] Setting key 58985ce096a328ea554a1f448f975892
I20260620 14:30:04.753962 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298: Bootstrap starting.
I20260620 14:30:04.756192 22508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:04.757022 22508 log.cc:824] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:04.758050 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.750222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58759 (local address 127.0.98.62:38889)
0620 14:30:04.750581 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.750599 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.750636 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:30:04.751460 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.751483 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.751877 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.752212 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:04.752572 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.753927 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.754728 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.755356 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.755501 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.756244 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:04.756266 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:04.756280 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:04.756731 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:04.756791 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:04.756823 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:04.756914 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.757057 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.757294 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260620 14:30:04.758823 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298: No bootstrap required, opened a new log
I20260620 14:30:04.760648 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:04.762712 22508 raft_consensus.cc:348] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e583deafa2fb4cf2b3b6d909d963d298" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } }
I20260620 14:30:04.762954 22508 raft_consensus.cc:374] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:04.763015 22508 raft_consensus.cc:729] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e583deafa2fb4cf2b3b6d909d963d298, State: Initialized, Role: FOLLOWER
I20260620 14:30:04.763464 22508 consensus_queue.cc:260] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e583deafa2fb4cf2b3b6d909d963d298" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } }
I20260620 14:30:04.763613 22508 raft_consensus.cc:388] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:04.763671 22508 raft_consensus.cc:482] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:04.763762 22508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:04.764582 22508 raft_consensus.cc:504] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e583deafa2fb4cf2b3b6d909d963d298" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } }
I20260620 14:30:04.764901 22508 leader_election.cc:302] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e583deafa2fb4cf2b3b6d909d963d298; no voters:
I20260620 14:30:04.765213 22508 leader_election.cc:288] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:04.765307 22518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:04.765533 22518 raft_consensus.cc:686] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [term 1 LEADER]: Becoming Leader. State: Replica: e583deafa2fb4cf2b3b6d909d963d298, State: Running, Role: LEADER
I20260620 14:30:04.765904 22518 consensus_queue.cc:237] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e583deafa2fb4cf2b3b6d909d963d298" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } }
I20260620 14:30:04.766414 22508 sys_catalog.cc:564] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:04.766548 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:04.762551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42251 (local address 127.0.98.62:38889)
0620 14:30:04.762659 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:04.762662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:04.762854 (+ 192us) server_negotiation.cc:407] Connection header received
0620 14:30:04.762938 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:04.762940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:04.762985 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:04.763071 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:04.763347 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.764026 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.764779 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:04.765122 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:04.765262 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:04.766040 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:04.766045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:04.766048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:04.766065 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:04.766090 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:04.766120 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:04.766165 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:04.766336 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 14:30:04.766384 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:30:04.767045 22518 sys_catalog.cc:455] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e583deafa2fb4cf2b3b6d909d963d298. Latest consensus state: current_term: 1 leader_uuid: "e583deafa2fb4cf2b3b6d909d963d298" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e583deafa2fb4cf2b3b6d909d963d298" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } } }
I20260620 14:30:04.767143 22518 sys_catalog.cc:458] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:04.767488 22525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:04.768314 22519 sys_catalog.cc:455] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e583deafa2fb4cf2b3b6d909d963d298" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e583deafa2fb4cf2b3b6d909d963d298" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38889 } } }
I20260620 14:30:04.768473 22519 sys_catalog.cc:458] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:04.770881 22525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:04.775410 22525 catalog_manager.cc:1347] Generated new cluster ID: 6ddd3c4d70fa43dba7c12dae3b19c898
I20260620 14:30:04.775468 22525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:04.796501 22525 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:04.797350 22525 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:04.803956 22525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e583deafa2fb4cf2b3b6d909d963d298: Generated new TSK 0
I20260620 14:30:04.804534 22525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:04.808005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 22440
2026-06-20T14:30:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:04 dist-test-slave-pk3j krb5kdc[22539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:04 dist-test-slave-pk3j krb5kdc[22539](info): set up 2 sockets
Jun 20 14:30:04 dist-test-slave-pk3j krb5kdc[22539](info): commencing operation
krb5kdc: starting...
W20260620 14:30:05.829183 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:05 dist-test-slave-pk3j krb5kdc[22539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:05.936856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:06.043622 22552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:06.043861 22552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:06.043934 22552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:06.047788 22552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:06.047863 22552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:06.047900 22552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:06.047951 22552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:06.047986 22552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:06.052902 22552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:06.053769 22552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:06.054652 22552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:06.062937 22560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:06.062956 22558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:06.062947 22557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:06.063215 22552 server_base.cc:1034] running on GCE node
I20260620 14:30:06.064004 22552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:06.065359 22552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:06.066558 22552 hybrid_clock.cc:648] HybridClock initialized: now 1781965806066534 us; error 38 us; skew 500 ppm
Jun 20 14:30:06 dist-test-slave-pk3j krb5kdc[22539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:06.069072 22552 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:06.069892 22552 webserver.cc:460] Webserver started at http://127.0.98.62:39359/ using document root <none> and password file <none>
I20260620 14:30:06.070390 22552 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:06.070472 22552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:06.070679 22552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:06.072579 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "b6656e8f90084866830362366a9daec9"
format_stamp: "Formatted at 2026-06-20 14:30:06 on dist-test-slave-pk3j"
server_key: "d7781d1053d2eccfb05a73ea8b50947c"
server_key_iv: "7fbf1807d08ff01547063bd72666968c"
server_key_version: "encryptionkey@0"
I20260620 14:30:06.073102 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6656e8f90084866830362366a9daec9"
format_stamp: "Formatted at 2026-06-20 14:30:06 on dist-test-slave-pk3j"
server_key: "d7781d1053d2eccfb05a73ea8b50947c"
server_key_iv: "7fbf1807d08ff01547063bd72666968c"
server_key_version: "encryptionkey@0"
I20260620 14:30:06.076362 22552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:30:06.078536 22567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:06.079562 22552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:30:06.079686 22552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "b6656e8f90084866830362366a9daec9"
format_stamp: "Formatted at 2026-06-20 14:30:06 on dist-test-slave-pk3j"
server_key: "d7781d1053d2eccfb05a73ea8b50947c"
server_key_iv: "7fbf1807d08ff01547063bd72666968c"
server_key_version: "encryptionkey@0"
I20260620 14:30:06.079784 22552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:06.083954 22552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:06.086540 22552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:06.086712 22552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:06.093472 22552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34775
I20260620 14:30:06.093495 22619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34775 every 8 connection(s)
I20260620 14:30:06.094254 22552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:06.096585 22620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:06.101559 22620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9: Bootstrap starting.
I20260620 14:30:06.101593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 22552
I20260620 14:30:06.101716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:06.101962 392 external_mini_cluster.cc:1428] Setting key fd52373a79f8c6e59a7059c0a17abe56
I20260620 14:30:06.103863 22620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:06.104528 22620 log.cc:824] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:06.106634 22620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9: No bootstrap required, opened a new log
I20260620 14:30:06.106719 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:06.103129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41285 (local address 127.0.98.62:34775)
0620 14:30:06.103526 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:06.103539 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:06.103571 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:30:06.104192 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:06.104208 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:06.104482 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:06.104785 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:06.105499 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:06.105504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:06.105517 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:06.105529 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:06.105590 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:06.105613 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:06.105718 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:06.105785 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:30:06.105913 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260620 14:30:06.109381 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:06.110625 22620 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6656e8f90084866830362366a9daec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34775 } }
I20260620 14:30:06.110837 22620 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:30:06.110908 22620 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6656e8f90084866830362366a9daec9, State: Initialized, Role: FOLLOWER
I20260620 14:30:06.111369 22620 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6656e8f90084866830362366a9daec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34775 } }
I20260620 14:30:06.111537 22620 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:30:06.111620 22620 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:30:06.111717 22620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:06.112586 22620 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6656e8f90084866830362366a9daec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34775 } }
I20260620 14:30:06.112924 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:06.111353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34755 (local address 127.0.98.62:34775)
0620 14:30:06.111455 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:06.111461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:06.111638 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:30:06.111769 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:06.111773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:06.111808 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:06.111889 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:06.112525 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:06.112531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:06.112537 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:30:06.112550 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:06.112577 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:30:06.112588 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:06.112685 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:06.112793 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:30:06.112824 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:30:06.112957 22620 leader_election.cc:302] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6656e8f90084866830362366a9daec9; no voters:
I20260620 14:30:06.113292 22620 leader_election.cc:288] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:30:06.113368 22630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:06.113535 22630 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [term 1 LEADER]: Becoming Leader. State: Replica: b6656e8f90084866830362366a9daec9, State: Running, Role: LEADER
I20260620 14:30:06.113804 22630 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6656e8f90084866830362366a9daec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34775 } }
I20260620 14:30:06.114468 22620 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:06.115387 22631 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6656e8f90084866830362366a9daec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6656e8f90084866830362366a9daec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34775 } } }
I20260620 14:30:06.115510 22631 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:06.115805 22632 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6656e8f90084866830362366a9daec9. Latest consensus state: current_term: 1 leader_uuid: "b6656e8f90084866830362366a9daec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6656e8f90084866830362366a9daec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34775 } } }
I20260620 14:30:06.115897 22632 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:06.115908 22639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:06.119009 22639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:06.124033 22639 catalog_manager.cc:1347] Generated new cluster ID: dfd73ab43e164d22b3345e16a2d0bbf5
I20260620 14:30:06.124091 22639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:06.133088 22639 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:06.133385 22639 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:06.140384 22639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6656e8f90084866830362366a9daec9: Generated new TSK 0
I20260620 14:30:06.140915 22639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:30:06.148309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 22552
2026-06-20T14:30:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1339 ms)
[----------] 18 tests from AuthTokenIssuingTest (24159 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:06 dist-test-slave-pk3j krb5kdc[22651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:06 dist-test-slave-pk3j krb5kdc[22651](info): set up 2 sockets
Jun 20 14:30:06 dist-test-slave-pk3j krb5kdc[22651](info): commencing operation
krb5kdc: starting...
W20260620 14:30:07.153396 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:07.259893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33205
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33205,127.0.98.61:36567,127.0.98.60:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:07.361335 22664 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:07.361611 22664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:07.361696 22664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:07.361778 22664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:07.365641 22664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:07.365710 22664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:07.365746 22664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:07.365801 22664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:07.365837 22664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:07.370785 22664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33205,127.0.98.61:36567,127.0.98.60:38759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:07.371680 22664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:07.372555 22664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:07.380225 22669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.380231 22670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:07.380288 22664 server_base.cc:1034] running on GCE node
W20260620 14:30:07.380232 22672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:07.380865 22664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:07.381835 22664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:07.383037 22664 hybrid_clock.cc:648] HybridClock initialized: now 1781965807383021 us; error 28 us; skew 500 ppm
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:07.385583 22664 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:07.386397 22664 webserver.cc:460] Webserver started at http://127.0.98.62:43887/ using document root <none> and password file <none>
I20260620 14:30:07.386888 22664 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:07.386970 22664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:07.387174 22664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:07.388995 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "9eaa5e86c49649e7a57c05ba360c2a54"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-pk3j"
server_key: "31796f8dd26cf2f9de8555fc88ac9370"
server_key_iv: "47e8fff37d0e272e70b7c58e7c89cb2e"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.389518 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "9eaa5e86c49649e7a57c05ba360c2a54"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-pk3j"
server_key: "31796f8dd26cf2f9de8555fc88ac9370"
server_key_iv: "47e8fff37d0e272e70b7c58e7c89cb2e"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.392902 22664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 14:30:07.395007 22679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.396042 22664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:07.396168 22664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "9eaa5e86c49649e7a57c05ba360c2a54"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-pk3j"
server_key: "31796f8dd26cf2f9de8555fc88ac9370"
server_key_iv: "47e8fff37d0e272e70b7c58e7c89cb2e"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.396263 22664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:07.422236 22664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:07.425081 22664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:07.425307 22664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:07.431681 22664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33205
I20260620 14:30:07.431926 22731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33205 every 8 connection(s)
I20260620 14:30:07.432510 22664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:07.434840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 22664
I20260620 14:30:07.434988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:07.435062 22732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:07.435267 392 external_mini_cluster.cc:1428] Setting key 1b5345a7f846d8d3f4af7fd6a286b95a
I20260620 14:30:07.438918 22732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:07.452886 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.440883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.0.98.62:33205)
0620 14:30:07.441240 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.441256 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.441595 (+ 339us) server_negotiation.cc:407] Connection header received
0620 14:30:07.442503 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.442507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.442660 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.442857 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.443648 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.444525 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.445851 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.446255 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.446432 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.448536 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.448547 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.448555 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.448573 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.450167 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.450602 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.450606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.450608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.450711 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.451007 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.451010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.451012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.451310 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.451431 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.451761 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.451970 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"spinlock_wait_cycles":1920,"thread_start_us":117,"threads_started":1}
I20260620 14:30:07.452895 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.440884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33205 (local address 127.0.0.1:53756)
0620 14:30:07.441453 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.441484 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.442238 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.442864 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.442869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.443248 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.443500 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.443520 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.444628 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.444631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.445730 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.445745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.446413 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.446417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.446504 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.447005 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.447020 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.448403 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.450268 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.450277 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.450285 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.450489 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.450800 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.450805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.450807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.450913 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.451419 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.451424 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.451547 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.451698 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.452023 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":92,"spinlock_wait_cycles":5504,"thread_start_us":148,"threads_started":1}
I20260620 14:30:07.455080 22732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:07.455687 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.455457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36567 (local address 127.0.0.1:54212)
0620 14:30:07.455547 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.455635 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:30:07.455863 22681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36567: connect: Connection refused (error 111)
I20260620 14:30:07.456595 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.456415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36567 (local address 127.0.0.1:54222)
0620 14:30:07.456506 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.456548 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:30:07.456732 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36567: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:30:07.458117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36567
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33205,127.0.98.61:36567,127.0.98.60:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:07.505945 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } attempt: 1
I20260620 14:30:07.506644 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.506384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36567 (local address 127.0.0.1:54226)
0620 14:30:07.506513 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.506574 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:30:07.507270 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.507073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36567 (local address 127.0.0.1:54228)
0620 14:30:07.507165 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.507214 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:30:07.507397 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36567: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36567: connect: Connection refused (error 111)
W20260620 14:30:07.559329 22739 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:07.559551 22739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:07.559572 22739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:07.559605 22739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:07.563333 22739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:07.563412 22739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:07.563433 22739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:07.563448 22739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:07.563468 22739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:07.568383 22739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33205,127.0.98.61:36567,127.0.98.60:38759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:07.569234 22739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:07.570106 22739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:07.575539 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } attempt: 2
I20260620 14:30:07.576386 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.576064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36567 (local address 127.0.0.1:54236)
0620 14:30:07.576208 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.576279 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:30:07.577261 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.576922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36567 (local address 127.0.0.1:54252)
0620 14:30:07.577016 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.577158 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:30:07.577419 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36567: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36567: connect: Connection refused (error 111)
W20260620 14:30:07.577768 22745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.577782 22744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.577816 22747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:07.578097 22739 server_base.cc:1034] running on GCE node
I20260620 14:30:07.578626 22739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:07.579582 22739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:07.580756 22739 hybrid_clock.cc:648] HybridClock initialized: now 1781965807580743 us; error 27 us; skew 500 ppm
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:07.583215 22739 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:30:07.583990 22739 webserver.cc:460] Webserver started at http://127.0.98.61:36731/ using document root <none> and password file <none>
I20260620 14:30:07.584499 22739 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:07.584581 22739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:07.584784 22739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:07.586705 22739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/data/instance:
uuid: "6baf3934f9f04956835e9a02b1bbefbe"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-pk3j"
server_key: "ee5a41c16078318193aea35146905433"
server_key_iv: "e15c6e38ecf9b0e1ce3f10c662a232d7"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.587224 22739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/wal/instance:
uuid: "6baf3934f9f04956835e9a02b1bbefbe"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-pk3j"
server_key: "ee5a41c16078318193aea35146905433"
server_key_iv: "e15c6e38ecf9b0e1ce3f10c662a232d7"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.590901 22739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:30:07.593142 22754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.594278 22739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:30:07.594408 22739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/wal
uuid: "6baf3934f9f04956835e9a02b1bbefbe"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-pk3j"
server_key: "ee5a41c16078318193aea35146905433"
server_key_iv: "e15c6e38ecf9b0e1ce3f10c662a232d7"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.594504 22739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:07.626667 22739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:07.629084 22739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:07.629285 22739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:07.635347 22739 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36567
I20260620 14:30:07.635597 22806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36567 every 8 connection(s)
I20260620 14:30:07.636168 22739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:30:07.638615 22807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:07.641809 22807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:07.643183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 22739
I20260620 14:30:07.643312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-1/wal/instance
I20260620 14:30:07.643579 392 external_mini_cluster.cc:1428] Setting key c4706beb4a521babb984897b6cba7e19
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 14:30:07.654824 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.643418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.0.98.62:33205)
0620 14:30:07.643546 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.643549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.644413 (+ 864us) server_negotiation.cc:407] Connection header received
0620 14:30:07.645409 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.645413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.645461 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.645530 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.646841 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.647742 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.648770 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.649198 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.649339 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.651575 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.651578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.651581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.651604 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.652851 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.653358 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.653361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.653362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.653433 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.653750 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.653754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.653755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.653998 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.654095 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.654630 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.654723 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:30:07.655678 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.643859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33205 (local address 127.0.0.1:53772)
0620 14:30:07.644275 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.644311 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.645172 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.645725 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.645735 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.646179 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.646694 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.646706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.647858 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.647861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.648637 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.648645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.649351 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.649354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.649476 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.650001 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.650017 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.651423 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.652952 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.652957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.652966 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.653280 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.653509 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.653511 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.653513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.653634 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.654103 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.654111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.654395 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.654591 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.654889 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":124,"threads_started":1}
I20260620 14:30:07.656950 22807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:07.666249 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.657390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36567 (local address 127.0.0.1:54260)
0620 14:30:07.657473 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.657486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.657554 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.658078 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.658081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.658241 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.658303 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.658307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.659181 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.659185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.659660 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.659666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.660189 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.660192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.660330 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.660764 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.660774 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.663161 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.664740 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.664743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.664744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.664952 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.665180 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.665182 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.665183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.665304 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.665729 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.665732 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.665859 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.665962 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.666048 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":31}
I20260620 14:30:07.666425 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.657522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54260 (local address 127.0.98.61:36567)
0620 14:30:07.657785 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.657795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.657818 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:30:07.657847 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.657850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.657969 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.658096 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.658401 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.659076 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.659762 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.660085 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.660192 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.663276 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.663283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.663291 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.663311 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.664632 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.665035 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.665039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.665040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.665107 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.665400 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.665402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.665403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.665589 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.665698 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.666021 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.666131 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":181,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:30:07.667795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38759
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33205,127.0.98.61:36567,127.0.98.60:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:07.668090 22807 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:07.668593 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } attempt: 3
I20260620 14:30:07.668740 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.668462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48826)
0620 14:30:07.668581 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.668652 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:30:07.668891 22755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
I20260620 14:30:07.669895 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.669705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48834)
0620 14:30:07.669820 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.669854 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 14:30:07.670069 22807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38759: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:07.677064 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.669309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.0.98.61:36567)
0620 14:30:07.669454 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.669458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.669469 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:07.669500 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.669502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.669531 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.669676 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.670170 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.670647 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.671925 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.672133 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.672213 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.674228 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.674231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.674232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.674251 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.675555 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.675981 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.675985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.675990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.676063 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.676342 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.676347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.676349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.676497 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.676593 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.676848 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.676944 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:30:07.677066 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.668963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36567 (local address 127.0.0.1:54272)
0620 14:30:07.669088 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.669103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.669180 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.669718 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.669723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.669919 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.670041 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.670049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.670798 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.670805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.671776 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.671790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.672264 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.672270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.672417 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.672813 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.672834 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.674085 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.675674 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.675679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.675683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.675888 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.676173 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.676176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.676178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.676258 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.676598 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.676603 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.676692 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.676790 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.676913 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:30:07.678110 22732 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:07.678671 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.678451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48840)
0620 14:30:07.678550 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.678596 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:30:07.679256 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.679056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48852)
0620 14:30:07.679176 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.679214 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 14:30:07.679405 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38759: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
I20260620 14:30:07.710685 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } attempt: 1
I20260620 14:30:07.711282 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.711045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48860)
0620 14:30:07.711161 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.711218 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:30:07.711957 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.711736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48874)
0620 14:30:07.711843 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.711904 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 14:30:07.712077 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38759: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
I20260620 14:30:07.719275 22807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } attempt: 1
I20260620 14:30:07.719826 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.719609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48878)
0620 14:30:07.719731 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.719775 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:30:07.720407 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.720239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48880)
0620 14:30:07.720325 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.720363 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:30:07.720546 22807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38759: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
W20260620 14:30:07.773082 22813 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:07.773396 22813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:07.773422 22813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:07.773458 22813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:07.777576 22813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:07.777622 22813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:07.777642 22813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:07.777657 22813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:07.777671 22813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:07.782446 22813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33205,127.0.98.61:36567,127.0.98.60:38759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:07.783259 22813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:07.784070 22813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:07.787231 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } attempt: 2
I20260620 14:30:07.788048 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.787726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48884)
0620 14:30:07.787915 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.787979 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 14:30:07.788764 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.788574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48890)
0620 14:30:07.788659 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.788717 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:30:07.788686 22807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } attempt: 2
W20260620 14:30:07.788938 22732 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38759: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
I20260620 14:30:07.789351 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.789104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48894)
0620 14:30:07.789231 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.789294 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:30:07.789964 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.789776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48904)
0620 14:30:07.789869 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.789910 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:30:07.790072 22807 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38759: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38759: connect: Connection refused (error 111)
I20260620 14:30:07.792179 22813 server_base.cc:1034] running on GCE node
W20260620 14:30:07.792148 22819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.792143 22822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:07.792148 22820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:07.792810 22813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:07.793769 22813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:07.794960 22813 hybrid_clock.cc:648] HybridClock initialized: now 1781965807794940 us; error 30 us; skew 500 ppm
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:07.797497 22813 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:30:07.798331 22813 webserver.cc:460] Webserver started at http://127.0.98.60:38741/ using document root <none> and password file <none>
I20260620 14:30:07.798861 22813 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:07.798954 22813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:07.799149 22813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:07.800968 22813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/data/instance:
uuid: "0e87709d6e9a4885836d099691bb961b"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-pk3j"
server_key: "cd479a6736dd25c7aadbd3e0e2e1ef32"
server_key_iv: "55d7c3ad437312c09b414375a135a60f"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.801487 22813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/wal/instance:
uuid: "0e87709d6e9a4885836d099691bb961b"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-pk3j"
server_key: "cd479a6736dd25c7aadbd3e0e2e1ef32"
server_key_iv: "55d7c3ad437312c09b414375a135a60f"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.804627 22813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:07.806701 22829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:07.807767 22813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:30:07.807885 22813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/wal
uuid: "0e87709d6e9a4885836d099691bb961b"
format_stamp: "Formatted at 2026-06-20 14:30:07 on dist-test-slave-pk3j"
server_key: "cd479a6736dd25c7aadbd3e0e2e1ef32"
server_key_iv: "55d7c3ad437312c09b414375a135a60f"
server_key_version: "encryptionkey@0"
I20260620 14:30:07.807988 22813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:07.824671 22813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:07.827185 22813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:07.827363 22813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:07.833628 22813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38759
I20260620 14:30:07.833905 22881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38759 every 8 connection(s)
I20260620 14:30:07.834434 22813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:30:07.837011 22882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:07.840121 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:07.843405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 22813
I20260620 14:30:07.843555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781965536974027-392-0/minicluster-data/master-2/wal/instance
I20260620 14:30:07.843861 392 external_mini_cluster.cc:1428] Setting key e76db04d1cf70fed80f1f9cac8cbc518
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:07.853103 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.842000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.0.98.62:33205)
0620 14:30:07.842106 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.842110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.842503 (+ 393us) server_negotiation.cc:407] Connection header received
0620 14:30:07.843406 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.843409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.843452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.843550 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.844914 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.845624 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.846809 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.847086 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.847163 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.849568 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.849571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.849572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.849592 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.850975 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.851621 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.851630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.851631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.851704 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.852056 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.852059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.852060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.852212 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.852282 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.852798 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.852922 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:30:07.853873 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.841996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33205 (local address 127.0.0.1:53778)
0620 14:30:07.842366 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.842400 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.843148 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.843761 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.843769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.844202 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.844765 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.844782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.845739 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.845745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.846685 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.846694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.847218 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.847222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.847413 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.847927 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.847944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.849439 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.851074 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.851084 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.851133 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.851504 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.851804 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.851807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.851809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.851964 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.852304 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.852313 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.852575 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.852743 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.852994 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":93,"threads_started":1}
I20260620 14:30:07.854040 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.844494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.0.98.60:38759)
0620 14:30:07.844840 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.844867 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.844892 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:30:07.845002 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.845007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.845174 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.845359 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.846073 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.846863 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.847892 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.848196 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.848279 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.850524 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.850529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.850537 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.850555 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.852354 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.852705 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.852712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.852715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.852798 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.853072 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.853075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.853077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.853403 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.853582 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.853724 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.853848 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260620 14:30:07.854933 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:07.862671 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.855305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36567 (local address 127.0.0.1:54276)
0620 14:30:07.855551 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.855562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.855666 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.855969 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.855971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.856131 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.856252 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.856259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.856947 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.856953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.857506 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.857513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.857947 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.857950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.858049 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.858481 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.858498 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.859709 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.861393 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.861397 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.861399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.861578 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.861835 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.861838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.861840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.861915 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.862247 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.862249 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.862333 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.862429 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.862534 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260620 14:30:07.862708 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.855393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.0.98.61:36567)
0620 14:30:07.855500 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.855505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.855634 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:30:07.855740 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.855743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.855787 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.855855 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.856349 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.856844 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.857632 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.857851 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.857948 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.859825 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.859828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.859830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.859850 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.861297 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.861666 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.861671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.861672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.861745 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.861995 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.861999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.862001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.862152 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.862252 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.862478 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.862578 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:30:07.863310 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:07.863556 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.855844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.0.98.62:33205)
0620 14:30:07.855932 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.855935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.856030 (+ 95us) server_negotiation.cc:407] Connection header received
0620 14:30:07.856103 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.856109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.856150 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.856228 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.856722 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.857173 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.857926 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.858141 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.858268 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.860492 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.860495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.860498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.860519 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.861908 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.862282 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.862285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.862287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.862358 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.862602 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.862607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.862608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.862822 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.863027 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.863315 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.863417 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:07.871605 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.863688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48924)
0620 14:30:07.863786 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.863801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.863869 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.864195 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.864198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.864344 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.864454 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.864461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.865402 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.865404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.866116 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.866125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.866692 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.866698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.866778 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.867173 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.867187 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.868515 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.870268 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.870272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.870276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.870523 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.870819 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.870825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.870830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.870921 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.871299 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.871301 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.871364 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.871438 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.871515 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 14:30:07.871791 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.863856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.0.98.60:38759)
0620 14:30:07.863936 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.863939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.863951 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:30:07.863982 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.863985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.864016 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.864095 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.864556 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.865310 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.866224 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.866581 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.866678 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.868633 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.868636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.868637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.868656 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.870160 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.870603 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.870607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.870609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.870717 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.871011 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.871016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.871019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.871211 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.871307 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.871512 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.871612 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:30:07.873247 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.864562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54288 (local address 127.0.98.61:36567)
0620 14:30:07.864635 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.864638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.864704 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:30:07.864829 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.864831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.864858 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.864915 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.865561 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.866219 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.867192 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.867542 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.867663 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.869966 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.869972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.869976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.870002 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.871619 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.872181 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.872183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.872184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.872251 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.872507 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.872510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.872512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.872667 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.872895 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.872995 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.873094 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:30:07.874965 22882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b: Bootstrap starting.
I20260620 14:30:07.876098 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:07.877311 22882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:07.877967 22882 log.cc:824] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:07.877247 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9eaa5e86c49649e7a57c05ba360c2a54"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53794
I20260620 14:30:07.879570 22882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b: No bootstrap required, opened a new log
I20260620 14:30:07.881191 22807 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } attempt: 3
I20260620 14:30:07.883174 22882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } }
I20260620 14:30:07.883452 22882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e87709d6e9a4885836d099691bb961b, State: Initialized, Role: FOLLOWER
I20260620 14:30:07.883733 22882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } }
I20260620 14:30:07.884132 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } } }
I20260620 14:30:07.884269 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:07.884913 22882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:07.888104 22732 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } attempt: 3
I20260620 14:30:07.889663 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.881559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48930)
0620 14:30:07.881645 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.881661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.881730 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.881942 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.881945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.882096 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.882160 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.882164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.883055 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.883058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.883612 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.883619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.884225 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.884229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.884316 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.884753 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.884767 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.886132 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.888001 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.888004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.888006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.888197 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.888733 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.888735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.888737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.888832 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.889255 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.889258 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.889339 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.889427 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.889533 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 14:30:07.890183 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.881609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.0.98.60:38759)
0620 14:30:07.881714 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.881718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.881730 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:30:07.881813 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.881816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.881849 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.881929 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.882263 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.882949 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.883713 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.884032 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.884119 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.886238 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.886241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.886244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.886262 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.887902 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.888534 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.888537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.888539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.888619 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.888930 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.888932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.888934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.889128 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.889887 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.889959 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.890054 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 14:30:07.890271 22899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:07.890329 22899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:07.888715 22813 master_runner.cc:186] Error getting master registration for 127.0.98.62:33205: OK, instance_id { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" instance_seqno: 1781965807429912 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:30:07.892084 22813 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:30:07.893126 22807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe: Bootstrap starting.
Jun 20 14:30:07 dist-test-slave-pk3j krb5kdc[22651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:07.895599 22807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:07.896412 22807 log.cc:824] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:07.898993 22807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe: No bootstrap required, opened a new log
I20260620 14:30:07.899309 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.888425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38759 (local address 127.0.0.1:48938)
0620 14:30:07.888543 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:07.888559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:07.888634 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:07.889137 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:07.889141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:07.889328 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:07.889420 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.889429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.892515 (+ 3086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.892518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.893346 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:07.893355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.893899 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.893902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:07.893996 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.894416 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:07.894457 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:07.895759 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:07.897867 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.897870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.897872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.898053 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.898342 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:07.898345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:07.898347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:07.898424 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:07.898825 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:07.898828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:07.898927 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 14:30:07.899038 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:30:07.899162 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:30:07.899349 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.888482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.0.98.60:38759)
0620 14:30:07.888847 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.888857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.888868 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:30:07.888918 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.888923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.888985 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.889536 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.889937 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.892404 (+ 2467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.893440 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.893806 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.893892 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.895898 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.895903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.895906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.895926 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.897782 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.898157 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.898161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.898163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.898264 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.898513 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.898518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.898520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.898695 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.898832 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.899096 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.899192 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":91,"threads_started":1}
I20260620 14:30:07.902040 22732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54: Bootstrap starting.
I20260620 14:30:07.902608 22807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } }
I20260620 14:30:07.902791 22807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6baf3934f9f04956835e9a02b1bbefbe, State: Initialized, Role: FOLLOWER
I20260620 14:30:07.903066 22807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } }
I20260620 14:30:07.903542 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } } }
I20260620 14:30:07.903746 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:07.904167 22807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:07.904184 22732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:07.904886 22732 log.cc:824] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54: Log is configured to *not* fsync() on all Append() calls
W20260620 14:30:07.908648 22915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:07.908741 22915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:07.907024 22739 master_runner.cc:186] Error getting master registration for 127.0.98.62:33205: OK, instance_id { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" instance_seqno: 1781965807429912 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:30:07.909971 22739 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:30:07.910576 22732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54: No bootstrap required, opened a new log
I20260620 14:30:07.912842 22732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } }
I20260620 14:30:07.913008 22732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eaa5e86c49649e7a57c05ba360c2a54, State: Initialized, Role: FOLLOWER
I20260620 14:30:07.913305 22732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } }
I20260620 14:30:07.913703 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } } }
I20260620 14:30:07.913861 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:07.914346 22732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:07.914443 22706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:30:07.914557 22706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:07.915608 22706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } }
I20260620 14:30:07.916442 22706 leader_election.cc:288] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [CANDIDATE]: Term 1 election: Requested vote from peers 6baf3934f9f04956835e9a02b1bbefbe (127.0.98.61:36567), 0e87709d6e9a4885836d099691bb961b (127.0.98.60:38759)
I20260620 14:30:07.917460 22856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e87709d6e9a4885836d099691bb961b"
I20260620 14:30:07.917677 22856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:07.918570 22856 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eaa5e86c49649e7a57c05ba360c2a54 in term 1.
I20260620 14:30:07.918962 22680 leader_election.cc:302] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e87709d6e9a4885836d099691bb961b, 9eaa5e86c49649e7a57c05ba360c2a54; no voters:
I20260620 14:30:07.919202 22917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:07.919402 22917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [term 1 LEADER]: Becoming Leader. State: Replica: 9eaa5e86c49649e7a57c05ba360c2a54, State: Running, Role: LEADER
I20260620 14:30:07.919667 22917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } }
I20260620 14:30:07.917145 22781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6baf3934f9f04956835e9a02b1bbefbe"
I20260620 14:30:07.921352 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eaa5e86c49649e7a57c05ba360c2a54. Latest consensus state: current_term: 1 leader_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } } }
I20260620 14:30:07.921401 22781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:07.921447 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:07.922664 22781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9eaa5e86c49649e7a57c05ba360c2a54 in term 1.
I20260620 14:30:07.926393 22664 master_runner.cc:263] No leader master found from master 9eaa5e86c49649e7a57c05ba360c2a54
I20260620 14:30:07.926457 22664 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:30:07.926580 22928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:07.926654 22685 tablet.cc:2369] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:30:07.927152 22928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:07.927315 22935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:07.927975 22935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:07.938881 22812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.926953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.0.98.61:36567)
0620 14:30:07.927067 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.927071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.930112 (+ 3041us) server_negotiation.cc:407] Connection header received
0620 14:30:07.930296 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.930299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.930344 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.930393 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.931333 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.932037 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.933182 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.933524 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.933599 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.935357 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.935361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.935364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.935387 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.937037 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.937576 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.937579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.937581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.937657 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.937950 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.937953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.937955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.938149 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.938285 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.938632 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.938732 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:30:07.949190 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.929888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.0.98.60:38759)
0620 14:30:07.929980 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.929984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.939297 (+ 9313us) server_negotiation.cc:407] Connection header received
0620 14:30:07.939454 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.939457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.939499 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.939548 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.940438 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.941090 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.942250 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.942609 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.942696 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.945463 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.945466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.945634 (+ 168us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.945662 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.947444 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.947927 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.947931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.947932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.948011 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.948307 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.948311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.948312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.948493 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.948610 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.948938 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.949047 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:30:07.951431 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:07.929559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.0.98.62:33205)
0620 14:30:07.929660 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:07.929664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:07.937439 (+ 7775us) server_negotiation.cc:407] Connection header received
0620 14:30:07.937606 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:07.937609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:07.937646 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:07.937689 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:07.938458 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.939164 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.940380 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:07.940661 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:07.940744 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:07.942278 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:07.942282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:07.942284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:07.942307 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:07.943958 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.949946 (+ 5988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.949949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.949951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.950029 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:07.950383 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:07.950386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:07.950387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:07.950593 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:07.950774 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:07.951163 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 14:30:07.951293 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:30:07.955036 22781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [term 1 FOLLOWER]: Refusing update from remote peer 9eaa5e86c49649e7a57c05ba360c2a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:07.955036 22856 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [term 1 FOLLOWER]: Refusing update from remote peer 9eaa5e86c49649e7a57c05ba360c2a54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:07.955621 22917 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:07.955806 22929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:07.959288 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eaa5e86c49649e7a57c05ba360c2a54. Latest consensus state: current_term: 1 leader_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } } }
I20260620 14:30:07.959409 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:07.959666 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eaa5e86c49649e7a57c05ba360c2a54. Latest consensus state: current_term: 1 leader_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } } }
I20260620 14:30:07.959767 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:07.969429 22917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } } }
I20260620 14:30:07.969532 22917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:07.970557 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } } }
I20260620 14:30:07.970676 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:07.971223 22942 mvcc.cc:205] Tried to move back new op lower bound from 7298931949357027328 to 7298931949242327040. Current Snapshot: MvccSnapshot[applied={T|T < 7298931949357027328}]
I20260620 14:30:07.971179 22929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } } }
I20260620 14:30:07.971289 22929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:07.971451 22938 mvcc.cc:205] Tried to move back new op lower bound from 7298931949357027328 to 7298931949242327040. Current Snapshot: MvccSnapshot[applied={T|T < 7298931949357027328}]
I20260620 14:30:07.972018 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9eaa5e86c49649e7a57c05ba360c2a54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33205 } } peers { permanent_uuid: "6baf3934f9f04956835e9a02b1bbefbe" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36567 } } peers { permanent_uuid: "0e87709d6e9a4885836d099691bb961b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38759 } } }
I20260620 14:30:07.972116 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:07.974133 22941 mvcc.cc:205] Tried to move back new op lower bound from 7298931949357027328 to 7298931949242327040. Current Snapshot: MvccSnapshot[applied={T|T < 7298931949357027328}]
I20260620 14:30:07.976451 22935 catalog_manager.cc:1347] Generated new cluster ID: 58ab3bbfd4df4dae85077242e4c4ab78
I20260620 14:30:07.976502 22935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:08.019160 22935 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:08.020392 22935 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:08.029615 22935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eaa5e86c49649e7a57c05ba360c2a54: Generated new TSK 0
I20260620 14:30:08.031047 22935 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:30:08.056883 22735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.054200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.0.98.62:33205)
0620 14:30:08.054314 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.054318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.055708 (+ 1390us) server_negotiation.cc:407] Connection header received
0620 14:30:08.056091 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.056094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.056140 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.056189 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.056768 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53818: BlockingRecv error: recv got EOF from 127.0.0.1:53818 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 14:30:08.057933 22900 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.055228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.0.98.60:38759)
0620 14:30:08.055354 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.055358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.056019 (+ 661us) server_negotiation.cc:407] Connection header received
0620 14:30:08.057233 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.057237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.057282 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.057332 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.057834 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48954: BlockingRecv error: recv got EOF from 127.0.0.1:48954 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260620 14:30:08.059290 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.054572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.0.98.61:36567)
0620 14:30:08.054670 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.054674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.058423 (+ 3749us) server_negotiation.cc:407] Connection header received
0620 14:30:08.058613 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.058616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.058658 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.058704 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.059191 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54320: BlockingRecv error: recv got EOF from 127.0.0.1:54320 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 14:30:08.063086 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.061765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.0.98.61:36567)
0620 14:30:08.061851 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.061855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.062287 (+ 432us) server_negotiation.cc:407] Connection header received
0620 14:30:08.062458 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.062461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.062521 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.062570 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.063007 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54324: BlockingRecv error: recv got EOF from 127.0.0.1:54324 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 14:30:08.067619 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.066245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.0.98.61:36567)
0620 14:30:08.066333 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.066337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.066857 (+ 520us) server_negotiation.cc:407] Connection header received
0620 14:30:08.067017 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.067020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.067062 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.067108 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.067540 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54336: BlockingRecv error: recv got EOF from 127.0.0.1:54336 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 14:30:08.076045 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.074718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.0.98.61:36567)
0620 14:30:08.074817 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.074821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.075285 (+ 464us) server_negotiation.cc:407] Connection header received
0620 14:30:08.075444 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.075447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.075488 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.075533 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.075967 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54348: BlockingRecv error: recv got EOF from 127.0.0.1:54348 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 14:30:08.083189 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.081863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.0.98.61:36567)
0620 14:30:08.081981 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.081984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.082451 (+ 467us) server_negotiation.cc:407] Connection header received
0620 14:30:08.082601 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.082604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.082643 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.082686 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.083115 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54358: BlockingRecv error: recv got EOF from 127.0.0.1:54358 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 14:30:08.094444 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.093041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.0.98.61:36567)
0620 14:30:08.093130 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.093134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.093643 (+ 509us) server_negotiation.cc:407] Connection header received
0620 14:30:08.093824 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.093827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.093867 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.093916 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.094353 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54360: BlockingRecv error: recv got EOF from 127.0.0.1:54360 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 14:30:08.113788 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.112597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.0.98.61:36567)
0620 14:30:08.112695 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.112699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.112955 (+ 256us) server_negotiation.cc:407] Connection header received
0620 14:30:08.113077 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.113080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.113131 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.113238 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.113682 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54370: BlockingRecv error: recv got EOF from 127.0.0.1:54370 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 14:30:08.148721 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.147890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.0.98.61:36567)
0620 14:30:08.147986 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.147989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.148051 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:30:08.148149 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.148152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.148189 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.148260 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.148610 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54378: BlockingRecv error: recv got EOF from 127.0.0.1:54378 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 14:30:08.215808 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.214631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.0.98.61:36567)
0620 14:30:08.214755 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.214771 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.214947 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:30:08.215127 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.215131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.215180 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.215264 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.215689 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54382: BlockingRecv error: recv got EOF from 127.0.0.1:54382 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 14:30:08.346778 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.345619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.0.98.61:36567)
0620 14:30:08.345771 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.345776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.345922 (+ 146us) server_negotiation.cc:407] Connection header received
0620 14:30:08.346083 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.346086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.346136 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.346217 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.346678 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54398: BlockingRecv error: recv got EOF from 127.0.0.1:54398 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 14:30:08.606155 22812 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:08.605028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.0.98.61:36567)
0620 14:30:08.605159 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:08.605164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:08.605301 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:30:08.605461 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:08.605465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:08.605520 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:08.605603 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:08.606048 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54402: BlockingRecv error: recv got EOF from 127.0.0.1:54402 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:30:08.891374 22899 catalog_manager.cc:1259] Loaded cluster ID: 58ab3bbfd4df4dae85077242e4c4ab78
I20260620 14:30:08.891438 22899 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b: loading cluster ID for follower catalog manager: success
I20260620 14:30:08.892942 22899 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b: acquiring CA information for follower catalog manager: success
I20260620 14:30:08.894003 22899 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e87709d6e9a4885836d099691bb961b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:30:08.909757 22915 catalog_manager.cc:1259] Loaded cluster ID: 58ab3bbfd4df4dae85077242e4c4ab78
I20260620 14:30:08.909814 22915 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe: loading cluster ID for follower catalog manager: success
I20260620 14:30:08.911289 22915 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe: acquiring CA information for follower catalog manager: success
I20260620 14:30:08.912314 22915 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6baf3934f9f04956835e9a02b1bbefbe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:30:09.123705 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:09.120035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.0.98.61:36567)
0620 14:30:09.120274 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:09.120278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:09.120297 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:30:09.120436 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:09.120439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:09.120572 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:09.120665 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:09.120915 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.121608 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.122433 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.122645 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.122718 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:09.122979 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:09.123408 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:09.123496 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:09.123544 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:30:09.123588 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
I20260620 14:30:09.129474 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:09.126365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.0.98.60:38759)
0620 14:30:09.126599 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:09.126603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:09.126616 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:30:09.126688 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:09.126690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:09.126803 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:09.126893 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:09.127059 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.127540 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.128250 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.128451 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.128521 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:09.128733 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:09.129074 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:09.129156 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:09.129244 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 14:30:09.129296 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":63,"threads_started":1}
I20260620 14:30:09.135623 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:09.131995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.0.98.61:36567)
0620 14:30:09.132104 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:09.132108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:09.132257 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:30:09.132443 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:09.132449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:09.132498 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:09.132561 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:09.132799 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.133528 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.134566 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.134813 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.134900 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:09.135189 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:09.135252 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:09.135322 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:09.135447 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:30:09.135501 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:30:09.135989 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:09.132527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.0.98.60:38759)
0620 14:30:09.132784 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:09.132788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:09.132838 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:30:09.132942 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:09.132946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:09.132985 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:09.133065 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:09.133375 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.133845 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.134792 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.135090 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.135180 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:09.135487 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:09.135562 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:09.135612 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:09.135834 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 14:30:09.135881 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260620 14:30:09.136901 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:09.131977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.0.98.62:33205)
0620 14:30:09.132259 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:09.132263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:09.132581 (+ 318us) server_negotiation.cc:407] Connection header received
0620 14:30:09.132675 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:09.132678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:09.132843 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:09.132965 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:09.133444 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.134212 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.135102 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:09.135453 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:09.135544 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:09.136167 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:09.136443 (+ 276us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:09.136488 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:09.136676 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:30:09.136740 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":62,"threads_started":1}
I20260620 14:30:09.144320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 22664
I20260620 14:30:09.150306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 22739
I20260620 14:30:09.156011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 22813
2026-06-20T14:30:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3009 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:30:09 dist-test-slave-pk3j krb5kdc[23030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:30:09 dist-test-slave-pk3j krb5kdc[23030](info): set up 2 sockets
Jun 20 14:30:09 dist-test-slave-pk3j krb5kdc[23030](info): commencing operation
krb5kdc: starting...
W20260620 14:30:10.169526 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/test-user.keytab.
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:30:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:30:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 14:30:10.273850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38277
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38277,127.0.98.61:37109,127.0.98.60:43269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:30:10.374629 23043 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:10.374884 23043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:10.374926 23043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:10.374982 23043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:10.378793 23043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:10.378834 23043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:10.378852 23043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:10.378867 23043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:10.378919 23043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:10.383569 23043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38277,127.0.98.61:37109,127.0.98.60:43269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:10.384383 23043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:10.385217 23043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:30:10.392520 23049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:10.392649 23043 server_base.cc:1034] running on GCE node
W20260620 14:30:10.392530 23048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:10.392530 23051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:10.393280 23043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:10.394207 23043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:10.395350 23043 hybrid_clock.cc:648] HybridClock initialized: now 1781965810395327 us; error 36 us; skew 500 ppm
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:10.397917 23043 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 14:30:10.398726 23043 webserver.cc:460] Webserver started at http://127.0.98.62:45475/ using document root <none> and password file <none>
I20260620 14:30:10.399192 23043 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:10.399245 23043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:10.399456 23043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:10.401103 23043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/data/instance:
uuid: "8240dc9fe24442c9b0751ed27007f1e6"
format_stamp: "Formatted at 2026-06-20 14:30:10 on dist-test-slave-pk3j"
server_key: "6f5f1629bbd4ef61848014e79a7230bb"
server_key_iv: "7cca29cc6d7e4d248ce3cfbfb7a437eb"
server_key_version: "encryptionkey@0"
I20260620 14:30:10.401648 23043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/wal/instance:
uuid: "8240dc9fe24442c9b0751ed27007f1e6"
format_stamp: "Formatted at 2026-06-20 14:30:10 on dist-test-slave-pk3j"
server_key: "6f5f1629bbd4ef61848014e79a7230bb"
server_key_iv: "7cca29cc6d7e4d248ce3cfbfb7a437eb"
server_key_version: "encryptionkey@0"
I20260620 14:30:10.404783 23043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:10.407078 23058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:10.408131 23043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:10.408262 23043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/wal
uuid: "8240dc9fe24442c9b0751ed27007f1e6"
format_stamp: "Formatted at 2026-06-20 14:30:10 on dist-test-slave-pk3j"
server_key: "6f5f1629bbd4ef61848014e79a7230bb"
server_key_iv: "7cca29cc6d7e4d248ce3cfbfb7a437eb"
server_key_version: "encryptionkey@0"
I20260620 14:30:10.408372 23043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:10.421590 23043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:10.424413 23043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:10.424582 23043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:10.430995 23043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38277
I20260620 14:30:10.431272 23110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38277 every 8 connection(s)
I20260620 14:30:10.431819 23043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/data/info.pb
I20260620 14:30:10.434237 23111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:10.437466 23111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:10.438357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 23043
I20260620 14:30:10.438477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-0/wal/instance
I20260620 14:30:10.438764 392 external_mini_cluster.cc:1428] Setting key 45753c0391fec54baeaa3ecdb0581a91
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 14:30:10.451197 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.439447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.0.98.62:38277)
0620 14:30:10.439820 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.439840 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.440088 (+ 248us) server_negotiation.cc:407] Connection header received
0620 14:30:10.440997 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.441002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.441163 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.441378 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.442114 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.442941 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.443863 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.444199 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.444436 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.446777 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.446782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.446791 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.446811 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.448360 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.448856 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.448864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.448869 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.449008 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.449343 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.449350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.449354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.449844 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.449944 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.450347 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.450511 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":239,"thread_start_us":116,"threads_started":1}
I20260620 14:30:10.451201 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.439448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38277 (local address 127.0.0.1:47630)
0620 14:30:10.439955 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.439987 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:10.440720 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:10.441376 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:10.441381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:10.441766 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:10.441982 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.441993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.443070 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.443073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.443747 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.443755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.444339 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.444342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.444519 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.445037 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:10.445051 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:10.446628 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:10.448467 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.448479 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.448490 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.448748 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.449106 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.449108 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.449109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.449257 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.449955 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:10.449973 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:10.450122 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 14:30:10.450294 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:30:10.450593 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":85,"spinlock_wait_cycles":27904,"thread_start_us":145,"threads_started":1}
I20260620 14:30:10.453336 23111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:10.453982 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.453711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37109 (local address 127.0.0.1:47086)
0620 14:30:10.453830 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.453909 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:30:10.454172 23059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37109: connect: Connection refused (error 111)
I20260620 14:30:10.454957 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.454718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37109 (local address 127.0.0.1:47090)
0620 14:30:10.454848 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.454895 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 14:30:10.455098 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37109: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37109: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 14:30:10.461838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37109
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38277,127.0.98.61:37109,127.0.98.60:43269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:10.504350 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } attempt: 1
I20260620 14:30:10.505146 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.504866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37109 (local address 127.0.0.1:47094)
0620 14:30:10.505005 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.505070 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:30:10.505890 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.505675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37109 (local address 127.0.0.1:47110)
0620 14:30:10.505799 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.505841 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:30:10.506083 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37109: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37109: connect: Connection refused (error 111)
W20260620 14:30:10.562932 23118 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:10.563172 23118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:10.563222 23118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:10.563292 23118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:10.567435 23118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:10.567485 23118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:10.567505 23118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:10.567521 23118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:10.567572 23118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:10.572531 23118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38277,127.0.98.61:37109,127.0.98.60:43269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:10.573446 23118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:10.574252 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } attempt: 2
I20260620 14:30:10.574370 23118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:10.575117 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.574770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37109 (local address 127.0.0.1:47116)
0620 14:30:10.574977 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.575046 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 14:30:10.575788 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.575596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37109 (local address 127.0.0.1:47126)
0620 14:30:10.575687 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.575728 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:30:10.575965 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37109: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37109: connect: Connection refused (error 111)
W20260620 14:30:10.582497 23123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:10.582522 23126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:10.582518 23124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:10.582644 23118 server_base.cc:1034] running on GCE node
I20260620 14:30:10.583407 23118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:10.584338 23118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:10.585508 23118 hybrid_clock.cc:648] HybridClock initialized: now 1781965810585491 us; error 30 us; skew 500 ppm
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:10.588037 23118 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 14:30:10.588793 23118 webserver.cc:460] Webserver started at http://127.0.98.61:38579/ using document root <none> and password file <none>
I20260620 14:30:10.589493 23118 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:10.589597 23118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:10.589799 23118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:10.591706 23118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/data/instance:
uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2"
format_stamp: "Formatted at 2026-06-20 14:30:10 on dist-test-slave-pk3j"
server_key: "89d0405f036fa3d06c117770dca729d0"
server_key_iv: "b3fce60570f1f86d43af6ec39cf769dd"
server_key_version: "encryptionkey@0"
I20260620 14:30:10.592221 23118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/wal/instance:
uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2"
format_stamp: "Formatted at 2026-06-20 14:30:10 on dist-test-slave-pk3j"
server_key: "89d0405f036fa3d06c117770dca729d0"
server_key_iv: "b3fce60570f1f86d43af6ec39cf769dd"
server_key_version: "encryptionkey@0"
I20260620 14:30:10.595590 23118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 14:30:10.597821 23133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:10.598870 23118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:10.599035 23118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/wal
uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2"
format_stamp: "Formatted at 2026-06-20 14:30:10 on dist-test-slave-pk3j"
server_key: "89d0405f036fa3d06c117770dca729d0"
server_key_iv: "b3fce60570f1f86d43af6ec39cf769dd"
server_key_version: "encryptionkey@0"
I20260620 14:30:10.599138 23118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:10.607753 23118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:10.610399 23118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:10.610570 23118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:10.616748 23118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37109
I20260620 14:30:10.617053 23185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37109 every 8 connection(s)
I20260620 14:30:10.617637 23118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/data/info.pb
I20260620 14:30:10.619988 23186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:10.623046 23186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:10.627017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 23118
I20260620 14:30:10.627108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-1/wal/instance
I20260620 14:30:10.627413 392 external_mini_cluster.cc:1428] Setting key a3fa6a75294589fa463b5d5af68d03fa
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 14:30:10.635461 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.624533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.0.98.62:38277)
0620 14:30:10.624653 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.624657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.625359 (+ 702us) server_negotiation.cc:407] Connection header received
0620 14:30:10.626099 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.626102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.626143 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.626215 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.627264 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.627820 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.629074 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.629352 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.629431 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.631934 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.631941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.631946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.631979 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.633547 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.633998 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.634003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.634007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.634095 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.634400 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.634404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.634406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.634644 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.634751 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.635148 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.635268 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:30:10.636015 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.624830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38277 (local address 127.0.0.1:47632)
0620 14:30:10.625225 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.625276 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:10.625943 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:10.626349 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:10.626355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:10.626675 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:10.627108 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.627129 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.627964 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.627972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.628942 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.628953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.629491 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.629497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.629702 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.630195 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:10.630217 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:10.631760 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:10.633660 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.633664 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.633675 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.633897 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.634193 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.634195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.634196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.634309 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.634745 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:10.634749 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:10.634929 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 14:30:10.635059 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:30:10.635258 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":99,"threads_started":1}
I20260620 14:30:10.637346 23186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:10.646559 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.637765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37109 (local address 127.0.0.1:47134)
0620 14:30:10.637957 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.637972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:10.638046 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:10.638443 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:10.638449 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:10.638632 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:10.638778 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.638787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.640155 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.640161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.641108 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.641122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.641711 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.641715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.641872 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.642249 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:10.642265 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:10.643490 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:10.645218 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.645221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.645222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.645387 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.645649 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.645651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.645653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.645724 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.646098 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:10.646101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:10.646170 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 14:30:10.646276 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:30:10.646404 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 14:30:10.646667 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.637879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.0.98.61:37109)
0620 14:30:10.638125 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.638139 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.638155 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:30:10.638188 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.638197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.638320 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.638416 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.638906 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.640049 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.641270 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.641595 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.641671 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.643610 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.643618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.643629 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.643649 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.645096 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.645486 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.645491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.645493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.645561 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.645802 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.645807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.645808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.645997 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.646117 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.646358 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.646485 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
I20260620 14:30:10.648155 23186 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:10.648721 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.648503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40488)
0620 14:30:10.648600 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.648656 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:30:10.648914 23135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
I20260620 14:30:10.649658 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.649487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40496)
0620 14:30:10.649575 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.649614 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:30:10.649849 23186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43269: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 14:30:10.651727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
/tmp/dist-test-taskDwUWGk/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43269
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38277,127.0.98.61:37109,127.0.98.60:43269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:30:10.667141 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } attempt: 3
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:10.675036 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.667611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.0.98.61:37109)
0620 14:30:10.667712 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.667718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.667733 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:30:10.667875 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.667878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.667934 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.668017 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.668393 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.668863 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.670022 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.670318 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.670428 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.672176 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.672179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.672181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.672199 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.673487 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.673915 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.673920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.673921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.673996 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.674272 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.674276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.674277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.674451 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.674550 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.674809 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.674906 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:30:10.675037 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.667555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37109 (local address 127.0.0.1:47136)
0620 14:30:10.667672 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.667687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:10.667771 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:10.668033 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:10.668036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:10.668195 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:10.668288 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.668293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.668973 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.668976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.669906 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.669915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.670429 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.670432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.670536 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.670965 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:10.670976 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:10.672048 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:10.673586 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.673590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.673592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.673810 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.674093 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.674096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.674098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.674180 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.674552 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:10.674557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:10.674650 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:30:10.674761 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:30:10.674885 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:30:10.676155 23111 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:10.676757 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.676533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40512)
0620 14:30:10.676647 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.676706 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:30:10.677413 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.677154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40518)
0620 14:30:10.677315 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.677359 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260620 14:30:10.677551 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43269: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
I20260620 14:30:10.699086 23186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } attempt: 1
I20260620 14:30:10.699697 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.699452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40526)
0620 14:30:10.699579 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.699632 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:30:10.700300 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.700099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40536)
0620 14:30:10.700193 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.700240 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:30:10.700418 23186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43269: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
I20260620 14:30:10.708694 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } attempt: 1
I20260620 14:30:10.709365 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.709103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40546)
0620 14:30:10.709242 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.709301 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:30:10.709977 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.709782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40550)
0620 14:30:10.709876 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.709924 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:30:10.710104 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43269: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
W20260620 14:30:10.753973 23193 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:30:10.754230 23193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:30:10.754288 23193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:30:10.754360 23193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:30:10.758224 23193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:30:10.758296 23193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:30:10.758332 23193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:30:10.758387 23193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:30:10.758422 23193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:30:10.763176 23193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38277,127.0.98.61:37109,127.0.98.60:43269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 13:47:06 UTC on ccycloud.quasar-lzmqik.root.comops.site
build id 4457504
I20260620 14:30:10.764015 23193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:30:10.764864 23193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:30:10.768507 23186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } attempt: 2
I20260620 14:30:10.769246 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.768976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40560)
0620 14:30:10.769096 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.769146 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:30:10.769845 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.769650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40564)
0620 14:30:10.769747 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.769791 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:30:10.769949 23186 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43269: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
W20260620 14:30:10.772746 23199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:10.772723 23198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:30:10.772738 23201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:30:10.773183 23193 server_base.cc:1034] running on GCE node
I20260620 14:30:10.773674 23193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:30:10.774523 23193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:30:10.775671 23193 hybrid_clock.cc:648] HybridClock initialized: now 1781965810775652 us; error 29 us; skew 500 ppm
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:30:10.778188 23193 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 14:30:10.778947 23193 webserver.cc:460] Webserver started at http://127.0.98.60:43519/ using document root <none> and password file <none>
I20260620 14:30:10.779433 23193 fs_manager.cc:359] Metadata directory not provided
I20260620 14:30:10.779517 23193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:30:10.779728 23193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:30:10.781567 23193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/data/instance:
uuid: "c3593b01fdff43e181fc975c8acf20aa"
format_stamp: "Formatted at 2026-06-20 14:30:10 on dist-test-slave-pk3j"
server_key: "ecabcabd6b5e028352611eb116646a2c"
server_key_iv: "50a85df81857a43a9f7c42fbfee9d1ea"
server_key_version: "encryptionkey@0"
I20260620 14:30:10.782089 23193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/wal/instance:
uuid: "c3593b01fdff43e181fc975c8acf20aa"
format_stamp: "Formatted at 2026-06-20 14:30:10 on dist-test-slave-pk3j"
server_key: "ecabcabd6b5e028352611eb116646a2c"
server_key_iv: "50a85df81857a43a9f7c42fbfee9d1ea"
server_key_version: "encryptionkey@0"
I20260620 14:30:10.785259 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } attempt: 2
I20260620 14:30:10.785458 23193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:30:10.785902 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.785634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40578)
0620 14:30:10.785738 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.785824 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:30:10.786510 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.786310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40588)
0620 14:30:10.786409 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.786453 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:30:10.786684 23111 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43269: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43269: connect: Connection refused (error 111)
I20260620 14:30:10.787810 23208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:30:10.789088 23193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:30:10.789252 23193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/wal
uuid: "c3593b01fdff43e181fc975c8acf20aa"
format_stamp: "Formatted at 2026-06-20 14:30:10 on dist-test-slave-pk3j"
server_key: "ecabcabd6b5e028352611eb116646a2c"
server_key_iv: "50a85df81857a43a9f7c42fbfee9d1ea"
server_key_version: "encryptionkey@0"
I20260620 14:30:10.789366 23193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:30:10.817482 23193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:30:10.819880 23193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:30:10.820061 23193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:30:10.825920 23193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43269
I20260620 14:30:10.826148 23260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43269 every 8 connection(s)
I20260620 14:30:10.826781 23193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/data/info.pb
I20260620 14:30:10.829141 23261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:30:10.832376 23261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:10.836596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu as pid 23193
I20260620 14:30:10.836702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDwUWGk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781965536974027-392-0/minicluster-data/master-2/wal/instance
I20260620 14:30:10.837002 392 external_mini_cluster.cc:1428] Setting key c681e097417428a9784b349b3c4e4006
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:10.844521 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.833850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.0.98.62:38277)
0620 14:30:10.833979 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.833983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.834656 (+ 673us) server_negotiation.cc:407] Connection header received
0620 14:30:10.835348 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.835351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.835392 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.835458 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.836415 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.837017 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.838391 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.838611 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.838717 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.841051 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.841054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.841056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.841075 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.842524 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.842981 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.842985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.842987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.843060 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.843366 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.843371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.843374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.843545 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.843649 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.844219 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.844338 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:30:10.845445 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.834170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38277 (local address 127.0.0.1:47648)
0620 14:30:10.834532 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.834565 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:10.835196 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:10.835589 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:10.835594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:10.835875 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:10.836281 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.836292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.837174 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.837178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.838258 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.838269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.838746 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.838749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.838897 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.839409 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:10.839427 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:10.840908 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:10.842644 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.842649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.842661 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.842886 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.843143 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.843147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.843149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.843273 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.843712 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:10.843717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:10.843969 (+ 252us) client_negotiation.cc:769] Sending connection context
0620 14:30:10.844152 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 14:30:10.844454 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260620 14:30:10.846753 23261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:10.848049 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.837621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.0.98.60:43269)
0620 14:30:10.837887 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.837899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.838095 (+ 196us) server_negotiation.cc:407] Connection header received
0620 14:30:10.838238 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.838250 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.838391 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.838547 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.839274 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.840234 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.841102 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.841479 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.841602 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.843826 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.843832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.843841 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.843862 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.845667 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.846206 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.846211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.846213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.846295 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.846546 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.846552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.846553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.846790 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.846985 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.847729 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.847858 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":103,"threads_started":1}
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 14:30:10.854833 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.847139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37109 (local address 127.0.0.1:47150)
0620 14:30:10.847248 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.847259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:10.847314 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:10.847661 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:10.847664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:10.847815 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:10.847929 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.847937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.848719 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.848721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.849168 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.849182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.849895 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.849899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.850013 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.850468 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:10.850481 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:10.851756 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:10.853500 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.853506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.853508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.853684 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.853951 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.853955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.853957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.854034 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.854437 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:10.854439 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:10.854516 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 14:30:10.854589 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:30:10.854698 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:30:10.854887 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.847308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.0.98.61:37109)
0620 14:30:10.847440 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.847445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.847464 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:30:10.847497 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.847499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.847534 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.847627 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.848033 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.848636 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.849481 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.849792 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.849889 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.851882 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.851887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.851895 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.851916 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.853403 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.853782 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.853787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.853789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.853864 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.854120 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.854125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.854126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.854324 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.854425 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.854671 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.854773 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:30:10.855597 23261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:30:10.857959 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.849690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.0.98.62:38277)
0620 14:30:10.849791 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.849794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.849804 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:30:10.849888 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.849890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.849930 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.850008 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.850481 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.851148 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.851981 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.852317 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.852430 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.854721 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.854724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.854726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.854748 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.856376 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.856783 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.856787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.856788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.856865 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.857129 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.857132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.857133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.857348 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.857547 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.857682 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.857788 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:10.861104 23186 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } attempt: 3
I20260620 14:30:10.863243 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.855996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40594)
0620 14:30:10.856081 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.856091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:10.856167 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:10.856374 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:10.856376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:10.856517 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:10.856567 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.856571 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.857291 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.857294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.857759 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.857766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.858361 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.858364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.858430 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.858843 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:10.858857 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:10.859992 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:10.861776 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.861778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.861780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.862006 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.862300 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.862305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.862307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.862386 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.862810 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:10.862813 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:10.862909 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 14:30:10.863012 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:30:10.863112 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:30:10.863387 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.856104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.0.98.60:43269)
0620 14:30:10.856202 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.856208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.856220 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:30:10.856250 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.856253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.856289 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.856364 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.856684 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.857146 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.857876 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.858187 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.858278 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.860108 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.860111 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.860114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.860136 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.861682 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.862114 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.862120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.862122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.862203 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.862481 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.862487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.862489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.862696 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.862798 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.863113 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.863244 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 14:30:10.866078 23261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa: Bootstrap starting.
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 14:30:10.868165 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.858982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.0.98.61:37109)
0620 14:30:10.859062 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.859066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.859119 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:30:10.859242 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.859244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.859273 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.859366 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.859866 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.860494 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.861161 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.861527 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.861633 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.864562 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.864566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.864568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.864589 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.866460 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.866881 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.866886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.866887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.866965 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.867275 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.867278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.867280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.867475 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.867628 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.867941 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.868031 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:30:10.868628 23261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:10.869510 23261 log.cc:824] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:10.870831 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.861445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40596)
0620 14:30:10.861736 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.861751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:10.861828 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:10.862037 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:10.862040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:10.862208 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:10.862301 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.862308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.864003 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.864008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.864787 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.864799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.865484 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.865488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.865565 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.865926 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:10.865942 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:10.867389 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:10.869442 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.869445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.869447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.869659 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.869927 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.869931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.869932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.870011 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.870422 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:10.870425 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:10.870501 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 14:30:10.870570 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:30:10.870684 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260620 14:30:10.871213 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:30:10.871405 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.861496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40596 (local address 127.0.98.60:43269)
0620 14:30:10.861736 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.861746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.861792 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:30:10.861893 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.861898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.861930 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.862026 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.862418 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.863884 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.864948 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.865370 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.865484 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.867516 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.867519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.867522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.867543 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.869327 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.869774 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.869777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.869778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.869852 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.870094 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.870099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.870100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.870316 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.871074 (+ 758us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.871141 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.871257 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":116,"threads_started":1}
I20260620 14:30:10.871529 23261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa: No bootstrap required, opened a new log
I20260620 14:30:10.873505 23186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2: Bootstrap starting.
I20260620 14:30:10.872390 23085 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8240dc9fe24442c9b0751ed27007f1e6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47656
I20260620 14:30:10.875303 23186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:10.875255 23261 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } }
I20260620 14:30:10.875463 23261 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3593b01fdff43e181fc975c8acf20aa, State: Initialized, Role: FOLLOWER
I20260620 14:30:10.875784 23261 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } }
I20260620 14:30:10.876070 23186 log.cc:824] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:10.876204 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } } }
I20260620 14:30:10.876322 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:10.876960 23261 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:10.880529 23186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2: No bootstrap required, opened a new log
W20260620 14:30:10.881685 23280 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:10.881755 23280 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:10.880342 23193 master_runner.cc:186] Error getting master registration for 127.0.98.62:38277: OK, instance_id { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" instance_seqno: 1781965810429192 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:30:10.883220 23193 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:30:10.885432 23186 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } }
I20260620 14:30:10.885627 23186 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6bd05c0eb1c47f0950249c9e8ae77b2, State: Initialized, Role: FOLLOWER
I20260620 14:30:10.885841 23111 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } attempt: 3
I20260620 14:30:10.885915 23186 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } }
I20260620 14:30:10.887099 23186 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:10.887671 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } } }
I20260620 14:30:10.888710 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 14:30:10 dist-test-slave-pk3j krb5kdc[23030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781965810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
W20260620 14:30:10.892149 23293 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:10.892263 23293 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:10.890245 23118 master_runner.cc:186] Error getting master registration for 127.0.98.62:38277: OK, instance_id { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" instance_seqno: 1781965810429192 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:30:10.893358 23118 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:30:10.895730 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.886166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43269 (local address 127.0.0.1:40612)
0620 14:30:10.886348 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:30:10.886360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:30:10.886427 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:30:10.887478 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:30:10.887481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:30:10.887656 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:30:10.887760 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.887770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.888695 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.888698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.889489 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:30:10.889498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.890155 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.890158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:30:10.890236 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.890707 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:30:10.890721 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:30:10.892118 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:30:10.894221 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.894226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.894228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.894460 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.894781 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:30:10.894784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:30:10.894786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:30:10.894873 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:30:10.895290 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:30:10.895293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:30:10.895373 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 14:30:10.895462 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:30:10.895577 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 14:30:10.896167 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.886241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.0.98.60:43269)
0620 14:30:10.887303 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.887307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.887319 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:30:10.887347 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.887350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.887389 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.887450 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.887877 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.888600 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.889618 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.889964 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.890054 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.892284 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.892287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.892291 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.892316 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.894101 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.894569 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.894572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.894574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.894663 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.894973 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.894976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.894978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.895166 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.895249 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.895946 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.896042 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994}
I20260620 14:30:10.898869 23111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6: Bootstrap starting.
I20260620 14:30:10.901139 23111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6: Neither blocks nor log segments found. Creating new log.
I20260620 14:30:10.901957 23111 log.cc:824] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:30:10.903801 23111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6: No bootstrap required, opened a new log
I20260620 14:30:10.906391 23111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } }
I20260620 14:30:10.906576 23111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8240dc9fe24442c9b0751ed27007f1e6, State: Initialized, Role: FOLLOWER
I20260620 14:30:10.906953 23111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } }
I20260620 14:30:10.907534 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } } }
I20260620 14:30:10.907790 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:10.908227 23111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:30:10.908263 23085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:30:10.908349 23085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:10.909408 23085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } }
I20260620 14:30:10.910701 23160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8240dc9fe24442c9b0751ed27007f1e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2"
I20260620 14:30:10.910918 23160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:10.912101 23160 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8240dc9fe24442c9b0751ed27007f1e6 in term 1.
I20260620 14:30:10.912480 23235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8240dc9fe24442c9b0751ed27007f1e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c3593b01fdff43e181fc975c8acf20aa"
I20260620 14:30:10.912681 23235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:30:10.913466 23059 leader_election.cc:302] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8240dc9fe24442c9b0751ed27007f1e6, a6bd05c0eb1c47f0950249c9e8ae77b2; no voters:
I20260620 14:30:10.910205 23085 leader_election.cc:288] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [CANDIDATE]: Term 1 election: Requested vote from peers a6bd05c0eb1c47f0950249c9e8ae77b2 (127.0.98.61:37109), c3593b01fdff43e181fc975c8acf20aa (127.0.98.60:43269)
I20260620 14:30:10.914391 23235 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8240dc9fe24442c9b0751ed27007f1e6 in term 1.
W20260620 14:30:10.916445 23307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:30:10.916514 23307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:30:10.919984 23043 master_runner.cc:263] No leader master found from master 8240dc9fe24442c9b0751ed27007f1e6
I20260620 14:30:10.920038 23043 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:30:10.920133 23296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:30:10.920337 23296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [term 1 LEADER]: Becoming Leader. State: Replica: 8240dc9fe24442c9b0751ed27007f1e6, State: Running, Role: LEADER
I20260620 14:30:10.920636 23296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } }
I20260620 14:30:10.922279 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8240dc9fe24442c9b0751ed27007f1e6. Latest consensus state: current_term: 1 leader_uuid: "8240dc9fe24442c9b0751ed27007f1e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } } }
I20260620 14:30:10.922415 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [sys.catalog]: This master's current role is: LEADER
W20260620 14:30:10.925838 23064 tablet.cc:2369] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:30:10.926105 23315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:30:10.926945 23315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:30:10.940186 23235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [term 1 FOLLOWER]: Refusing update from remote peer 8240dc9fe24442c9b0751ed27007f1e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:10.940804 23313 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:10.943980 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.922946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.0.98.60:43269)
0620 14:30:10.923040 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.923044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.929445 (+ 6401us) server_negotiation.cc:407] Connection header received
0620 14:30:10.929607 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.929610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.929650 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.929708 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.930614 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.931352 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.932504 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.932840 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.932903 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.936840 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.936843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.936846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.936869 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.938549 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.939045 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.939048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.939050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.939126 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.943115 (+ 3989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.943118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.943120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.943313 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.943406 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.943737 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.943842 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:30:10.943982 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8240dc9fe24442c9b0751ed27007f1e6. Latest consensus state: current_term: 1 leader_uuid: "8240dc9fe24442c9b0751ed27007f1e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } } }
I20260620 14:30:10.944401 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:10.944762 23160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [term 1 FOLLOWER]: Refusing update from remote peer 8240dc9fe24442c9b0751ed27007f1e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:30:10.945271 23313 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:30:10.948266 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8240dc9fe24442c9b0751ed27007f1e6. Latest consensus state: current_term: 1 leader_uuid: "8240dc9fe24442c9b0751ed27007f1e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } } }
I20260620 14:30:10.948367 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:10.952642 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8240dc9fe24442c9b0751ed27007f1e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } } }
I20260620 14:30:10.952750 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:10.953015 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8240dc9fe24442c9b0751ed27007f1e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } } }
I20260620 14:30:10.953110 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:10.953475 23320 mvcc.cc:205] Tried to move back new op lower bound from 7298931961604608000 to 7298931961534349312. Current Snapshot: MvccSnapshot[applied={T|T < 7298931961604608000}]
I20260620 14:30:10.954427 23319 mvcc.cc:205] Tried to move back new op lower bound from 7298931961604608000 to 7298931961534349312. Current Snapshot: MvccSnapshot[applied={T|T < 7298931961604608000}]
I20260620 14:30:10.957412 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.924067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.0.98.61:37109)
0620 14:30:10.924161 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.924165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.934954 (+ 10789us) server_negotiation.cc:407] Connection header received
0620 14:30:10.935173 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.935176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.935218 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.935265 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.939422 (+ 4157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.940144 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.946782 (+ 6638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.947120 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.947190 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.951626 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.951630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.951633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.951656 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.954305 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.955240 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.955243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.955245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.955320 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.955946 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.955949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.955951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.956133 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.956247 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.957146 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.957270 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:30:10.959045 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:10.923554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.0.98.62:38277)
0620 14:30:10.923640 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:10.923645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:10.934782 (+ 11137us) server_negotiation.cc:407] Connection header received
0620 14:30:10.935996 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:10.936000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:10.936041 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:10.936086 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:10.940385 (+ 4299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.941114 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.945801 (+ 4687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:10.946122 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:10.946189 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:10.949867 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:30:10.949870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:30:10.949873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:30:10.949896 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:30:10.951473 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.955586 (+ 4113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.955590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.955591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.955665 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:30:10.956508 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:30:10.956512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:30:10.956513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:30:10.956743 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:30:10.956842 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:10.958809 (+ 1967us) server_negotiation.cc:299] Negotiation successful
0620 14:30:10.958915 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:30:10.961706 23315 catalog_manager.cc:1347] Generated new cluster ID: 92369f52dea34ed7b8692ea6642c1288
I20260620 14:30:10.961758 23315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:30:10.964036 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8240dc9fe24442c9b0751ed27007f1e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } } }
I20260620 14:30:10.964135 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:30:10.964990 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8240dc9fe24442c9b0751ed27007f1e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8240dc9fe24442c9b0751ed27007f1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38277 } } peers { permanent_uuid: "a6bd05c0eb1c47f0950249c9e8ae77b2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37109 } } peers { permanent_uuid: "c3593b01fdff43e181fc975c8acf20aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43269 } } }
I20260620 14:30:10.965078 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:30:10.965677 23322 mvcc.cc:205] Tried to move back new op lower bound from 7298931961604608000 to 7298931961534349312. Current Snapshot: MvccSnapshot[applied={T|T < 7298931961604608000}]
I20260620 14:30:10.974509 23315 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:30:10.976966 23315 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:30:10.990231 23315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8240dc9fe24442c9b0751ed27007f1e6: Generated new TSK 0
I20260620 14:30:10.990932 23315 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:30:11.023381 23266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.020400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40638 (local address 127.0.98.60:43269)
0620 14:30:11.020506 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.020510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.022426 (+ 1916us) server_negotiation.cc:407] Connection header received
0620 14:30:11.022606 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.022609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.022652 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.022714 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.023271 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40638: BlockingRecv error: recv got EOF from 127.0.0.1:40638 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 14:30:11.024725 23113 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.021062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.0.98.62:38277)
0620 14:30:11.021156 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.021160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.023937 (+ 2777us) server_negotiation.cc:407] Connection header received
0620 14:30:11.024089 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.024093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.024133 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.024178 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.024617 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47676: BlockingRecv error: recv got EOF from 127.0.0.1:47676 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 14:30:11.026047 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.021657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.0.98.61:37109)
0620 14:30:11.021827 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.021831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.025225 (+ 3394us) server_negotiation.cc:407] Connection header received
0620 14:30:11.025389 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.025393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.025439 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.025489 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.025948 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47174: BlockingRecv error: recv got EOF from 127.0.0.1:47174 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 14:30:11.030041 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.028639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.0.98.61:37109)
0620 14:30:11.028742 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.028747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.029120 (+ 373us) server_negotiation.cc:407] Connection header received
0620 14:30:11.029331 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.029335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.029404 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.029450 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.029940 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47178: BlockingRecv error: recv got EOF from 127.0.0.1:47178 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 14:30:11.034590 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.033345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.0.98.61:37109)
0620 14:30:11.033431 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.033435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.033851 (+ 416us) server_negotiation.cc:407] Connection header received
0620 14:30:11.034009 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.034012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.034052 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.034098 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.034510 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47188: BlockingRecv error: recv got EOF from 127.0.0.1:47188 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 14:30:11.039826 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.038554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.0.98.61:37109)
0620 14:30:11.038638 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.038642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.039054 (+ 412us) server_negotiation.cc:407] Connection header received
0620 14:30:11.039221 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.039224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.039266 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.039313 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.039756 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47196: BlockingRecv error: recv got EOF from 127.0.0.1:47196 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 14:30:11.047221 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.045873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.0.98.61:37109)
0620 14:30:11.045980 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.045983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.046397 (+ 414us) server_negotiation.cc:407] Connection header received
0620 14:30:11.046568 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.046571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.046613 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.046703 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.047147 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47204: BlockingRecv error: recv got EOF from 127.0.0.1:47204 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 14:30:11.058703 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.057370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.0.98.61:37109)
0620 14:30:11.057459 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.057463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.057901 (+ 438us) server_negotiation.cc:407] Connection header received
0620 14:30:11.058084 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.058087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.058129 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.058178 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.058614 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47214: BlockingRecv error: recv got EOF from 127.0.0.1:47214 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 14:30:11.078562 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.077092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.0.98.61:37109)
0620 14:30:11.077229 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.077234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.077716 (+ 482us) server_negotiation.cc:407] Connection header received
0620 14:30:11.077928 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.077931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.077975 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.078021 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.078471 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47226: BlockingRecv error: recv got EOF from 127.0.0.1:47226 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 14:30:11.113919 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.112798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.0.98.61:37109)
0620 14:30:11.112919 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.112923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.113092 (+ 169us) server_negotiation.cc:407] Connection header received
0620 14:30:11.113260 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.113267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.113311 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.113394 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.113796 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47234: BlockingRecv error: recv got EOF from 127.0.0.1:47234 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 14:30:11.181020 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.179960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.0.98.61:37109)
0620 14:30:11.180135 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.180139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.180222 (+ 83us) server_negotiation.cc:407] Connection header received
0620 14:30:11.180325 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.180328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.180372 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.180443 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.180904 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47248: BlockingRecv error: recv got EOF from 127.0.0.1:47248 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 14:30:11.312216 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.311038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.0.98.61:37109)
0620 14:30:11.311164 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.311170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.311355 (+ 185us) server_negotiation.cc:407] Connection header received
0620 14:30:11.311535 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.311538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.311586 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.311673 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.312114 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47254: BlockingRecv error: recv got EOF from 127.0.0.1:47254 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 14:30:11.571368 23190 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:30:11.570338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.0.98.61:37109)
0620 14:30:11.570463 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:11.570467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:11.570570 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:30:11.570720 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:11.570722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:11.570768 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:11.570836 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:30:11.571233 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47262: BlockingRecv error: recv got EOF from 127.0.0.1:47262 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:30:11.882863 23280 catalog_manager.cc:1259] Loaded cluster ID: 92369f52dea34ed7b8692ea6642c1288
I20260620 14:30:11.882927 23280 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa: loading cluster ID for follower catalog manager: success
I20260620 14:30:11.884413 23280 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa: acquiring CA information for follower catalog manager: success
I20260620 14:30:11.885577 23280 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3593b01fdff43e181fc975c8acf20aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:30:11.893397 23293 catalog_manager.cc:1259] Loaded cluster ID: 92369f52dea34ed7b8692ea6642c1288
I20260620 14:30:11.893447 23293 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2: loading cluster ID for follower catalog manager: success
I20260620 14:30:11.894780 23293 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2: acquiring CA information for follower catalog manager: success
I20260620 14:30:11.895668 23293 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6bd05c0eb1c47f0950249c9e8ae77b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:30:12.090150 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:12.085794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.0.98.61:37109)
0620 14:30:12.086076 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:12.086082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:12.086097 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:30:12.086156 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:12.086160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:12.086319 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:12.086449 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:12.086800 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:12.087615 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:12.088521 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:12.088842 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:12.088955 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:12.089305 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:12.089732 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:12.089869 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:12.089928 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:30:12.089984 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
I20260620 14:30:12.096391 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:12.092917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.0.98.60:43269)
0620 14:30:12.093149 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:12.093154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:12.093167 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:30:12.093332 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:12.093335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:12.093452 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:12.093578 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:12.093756 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:12.094297 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:12.095076 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:12.095292 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:12.095420 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:12.095681 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:12.096020 (+ 339us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:12.096145 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:12.096185 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:30:12.096230 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260620 14:30:12.103170 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:12.099021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.0.98.60:43269)
0620 14:30:12.099117 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:12.099121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:12.099257 (+ 136us) server_negotiation.cc:407] Connection header received
0620 14:30:12.099484 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:12.099490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:12.099544 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:12.099608 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:12.099873 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:12.100492 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:12.101588 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:12.101971 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:12.102081 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:12.102518 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:12.102600 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:12.102672 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:12.102980 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 14:30:12.103037 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:30:12.104203 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:12.099502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.0.98.61:37109)
0620 14:30:12.099614 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:12.099618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:12.099759 (+ 141us) server_negotiation.cc:407] Connection header received
0620 14:30:12.099926 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:12.099929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:12.099983 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:12.100052 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:12.100278 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:12.101049 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:12.102457 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:12.102762 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:12.102854 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:12.103655 (+ 801us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:12.103742 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:12.103832 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:12.103991 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:30:12.104050 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:30:12.104467 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:30:12.098983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.0.98.62:38277)
0620 14:30:12.099199 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:30:12.099203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:30:12.099543 (+ 340us) server_negotiation.cc:407] Connection header received
0620 14:30:12.099625 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:30:12.099628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:30:12.099766 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:30:12.099868 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:30:12.100215 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:12.100979 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:12.102444 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:30:12.102774 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:30:12.102858 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:30:12.103524 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:30:12.103831 (+ 307us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:30:12.103912 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:30:12.104046 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:30:12.104311 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260620 14:30:12.107944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 23043
I20260620 14:30:12.114554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 23118
I20260620 14:30:12.120285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDwUWGk/build/debug/bin/kudu with pid 23193
2026-06-20T14:30:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2964 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5973 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275140 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST